The SharePoint / Perl Connection (Part 1)

Why on earth would these two disparate camps ever meet? Perl is 15 million years old, right, and SharePoint is the bane to system administrators everywhere. The small “SharePerl” audience includes me, and two or three really old techs that still use land line phones.

But Perl can make a nice replacement for SharePoint Timer Jobs and can be easier to debug than a Visual Studio developed solution. (Okay, whatever… My .NET experience is only deep enough that I can spell it.)

Setup

I have ActivePerl installed on an XP VM, which I use for all my script development. ActivePerl comes with a Perl Package Manager, which allows you to install any one of 12,000 Perl MODs with relative ease. Over the years I’ve installed a good number of these MODs ranging from simulating poker hands to creating PDF files script-o-matically. When IE finally supports SVG, my cachet will rise dynamically, thanks to Perl MODs.

Perl does not need to be installed on a SharePoint server. Don’t do that. That’s crazy. Perl can be run from any desktop, with the caveat: If you want to schedule Perl scripts to run at certain times, the machine has to be on, with a network connection. The interaction from Perl to SharePoint happens thanks to SharePoint web services and Perl’s LWP MOD.

LWP

LWP stands for “little www-browser for Perl”. It’s a collection of functions and classes that allow Perl to become a simulated web-browser. Ever wonder how email scraping robots work? I think I can safely say that LWP is involved. LWP is in the hacker tool category of “fun, fun, fun”. Go find a poll or survey that doesn’t restrict multiple entries by IP address, and script up an iterative LWP session. Okay, there’s still a law about self-incrimination, right?

LWP uses a Perl site level authentication library specifically tailored for NTLM:

C:\Perl\site\lib\Authen\NTLM.pm

Unfortunately, the library used by LWP for NTLM authentication is not compatible with IIS6 or 7. NTLM.pm needs to be hacked to send parts of the authentication packet in Unicode:

Edit NTLM.pm and find the line:

$domain = substr($challenge, $c_info->{domain}{offset}, $c_info->{domain}{len});

and replace that code with (or better yet, comment out the code above and insert the code below):

$domain = &unicode($domain);

SOAP::Lite

SOAP::Lite is a MOD that is subtitled, “Perl's Web Services Toolkit”. If you’ve ever had the privilege of working with SharePoint’s Web Services, you’ll know they can handle almost anything you’ve ever wanted to do with a SharePoint site. This functionality is accomplished via SOAP calls.

When you load a web part page into SharePoint Designer (2007), have you ever seen the SOAP calls churning in the bottom left hand corner?

We’ll use this MOD to formulate SOAP calls to our SharePoint Web Services.

Credit, Where Due

Most of what I learned about the SharePoint / Perl connection, I got from a blog post from a Londoner named James who blogs at WWW.SQUISH.NET. His post entitled, “Talking to SharePoint Lists with Perl” is a gem. James’ bio mentioned he was working in banking, and that he loves Lotus Notes. Well, God bless him.

Let’s Roll

I’m going to just paste the Perl code here:

  1: use LWP::UserAgent;
  2: use LWP::Debug;
  3: use Data::Dumper;
  4: use SOAP::Lite on_action => sub { "$_[0]$_[1]"; };
  5: import SOAP::Data 'name', 'value';
  6: our $sp_endpoint = 'http://shareperl/sites/wef/_vti_bin/lists.asmx';
  7: our $sp_domain = 'shareperl:80';
  8: our $sp_username = 'PIMPS\\wef';
  9: our $sp_password = 'Pa$$word16';
 10:
 11: $debug = 0;
 12:
 13: if ($debug) {
 14:     LWP::Debug::level('+');
 15:     SOAP::Lite->import(+trace => 'all');
 16: }
 17:
 18: my @ua_args = (keep_alive => 1);
 19: my @credentials = ($sp_domain, "", $sp_username, $sp_password);
 20: my $schema_ua = LWP::UserAgent->new(@ua_args);
 21: $schema_ua->credentials(@credentials);
 22: $soap = SOAP::Lite->proxy($sp_endpoint, @ua_args, credentials => \@credentials);
 23: $soap->schema->useragent($schema_ua);
 24: $soap->uri("http://schemas.microsoft.com/sharepoint/soap/");
 25:
 26: $lists = $soap->GetListCollection();
 27: quit(1, $lists->faultstring()) if defined $lists->fault();
 28:
 29: sub lists_getid
 30: {
 31:     my $title = shift;
 32:     my @result = $lists->dataof('//GetListCollectionResult/Lists/List');
 33:     foreach my $data (@result) {
 34:         my $attr = $data->attr;
 35:         return $attr->{ID} if ($attr->{Title} eq $title);
 36:     }
 37:     return undef;
 38: }
 39:
 40: sub lists_getitems
 41: {
 42:     my $listid = shift;
 43:     my $in_listName = name('listName' => $listid);
 44:     my $in_viewName = name('viewName' => '');
 45:     my $in_rowLimit = name('rowLimit' => 99999);
 46:     my $call = $soap->GetListItems($in_listName, $in_viewName, $in_rowLimit);
 47:     quit(1, $call->faultstring()) if defined $call->fault();
 48:     return $call->dataof('//GetListItemsResult/listitems/data/row');
 49: }
 50:
 51: my $list_id = lists_getid('Disk Space');
 52: print "List ID is: $list_id\n";
 53: my @items = lists_getitems($list_id);
 54: foreach my $data (@items) {
 55:     my $attr = $data->attr;
 56:     print Dumper($attr);
 57: }

Here’s the output from the command-line:



And here’s the view of the SharePoint list:


Much more to come in later posts, but I hope you get an idea of where you can go using Perl to interface with a SharePoint list…

SharePoint Designer (2007) Best Advice

"From hell's heart I stab at thee!" (from "Moby Dick" geeks! Gosh, read a book that isn't the Start Trek Klingon Dictionary)

There's a lot of basic advice about getting SharePoint Designer (SPD) to behave properly:

• Make sure your computer has 512 TB of RAM
• Keep your OS and MSO patched
• Wash your coffee pots out with a little CLR now and again.

After much consternation and quiet sobbing, here are some tips on getting SPD to do what you want in a group development situation:

Work on files (web part pages, etc.) stored in a Document Library that has both "versioning" and "require files to be checked out before editing" enabled. Check out will keep files locked for your own amusement only. You must check in files for others to see them, but then you eliminate the "last in, wins" scenario. Check in early and often, each check in creates a new version. Describe these versions with details, so that, if you need to recover after a mistake, you can restore an older version without needing to dig into a laborious site restore. (You backup every twelve seconds to a .cmp file, right?)

Next, only work on one file at a time. SPD works with SOAP calls to web parts, which are stored in a SharePoint database. It will cache everything you need locally until you save and check in a file. A great metaphor for working with files in SPD is that it is similar to driving a Mars Rover.

Quick Exercise:

• Try driving multiple remote controlled cars on your dining room floor.
  
• Next, try driving multiple remote controlled vehicles millions of miles away on Mars.

My last tip, and this is for cool middle tier devs working in loft-style offices: If you are editing in SPD with an iffy wireless connection... Sit at a wired connection to your dev farm!




"N*** please, I'm the macaroni and the cheese!" --Puff Daddy, pre-Diddy

Developing for the Middle Tier

I've been busy lately developing a LARGE solution for a client with practically no hope of gaining console or CA access. This pretty much relegates my efforts to developing in SharePoint's middle tier, so that no solutions need to be packaged, no code goes into the GAC, and no VS used. This means scripting vs. coding. As an old Perl hacker, this is fun to me. We've been able to rapidly develop and deploy a complex "no-code" solution in SharePoint.

But, can I really call myself a SharePoint "developer"? I don't think so, but maybe. In any case, let's review the SharePoint's "Development" Tiers:

Bottom/Tier 1--COTS, aka OOB or End User. Can you click "Site Actions" -> "Edit Page"? You are a SharePoint developer now!

Middle/Tier 2--Scripting using SharePoint Designer (SPD). Tons of effort goes in here, as well as patience. SPD is free, but it isn't free. It's the most frustrating software I've ever used. Learn DHTML/XML/XSLT/CSS/Master Pages/JQuery/Web Services.

Top/Tier 3--Coding and packaging SharePoint solutions in Visual Studio. This is the true "developer" tier. If you are here, you are in great demand (MOSS 2007) and probably don't have an 80 BELF Paladin.

SharePoint Deployment SWOT Analysis

I have a reason for throwing SharePoint down a well-defined method of business analysis. This has become increasingly more important, as I have become some kind of SharePoint "huckster", with no innate talent for sales.

So, here I go, a SWOT analysis of SharePoint. SWOT is really for business firm analysis, so this is a bit of a stretch.

Strengths:

Microsoft Enterprise level product
Nine products in one package (OOB)
Web-based tool eliminates the need for VPN
Deployed in many diverse corporate environs

Weaknesses:

Microsoft brand is tarnished
Expensive upfront and per user
Requires a knowledgeable executive-level champion

Opportunities:

Organize and find business documents fast
Improve business processes and the flow of information
Extensible
Rapid solution deployment
Can help companies become more transparent

Threats:

Scope creep can render solutions inoperable
Lack of governance and planning can lead to a failed deployment

SharePoint Jobs Demystified

The "SharePoint" job market seems to be bustling, despite the economic woes. Perhaps companies that invested in the product are now trying to poke at that ROI that Microsoft promised at the beginning. Companies need to make changes to improve their processes, in order to save money, and to play catchup as they jump headlong into the Data Age.

I'm pretty sure that SharePoint jobs come in three categories:

"SharePoint Developer"

Oh happy day, if you have 5 years of .NET development experience with C# -and- you know the SharePoint 2007 Object Model or API. There are a plethora of job ops out there for someone with these skills, but be wary: Companies without a well defined set of objectives for their SharePoint development solution will be tough to work for.

For the most part, SharePoint devs will be working with large data sets outside of SharePoint. What SharePoint does with this data falls into two categories: query and display or convert into a SharePoint structure. Gosh it sounds like Perl!

"SharePoint Architect"

Here, they need someone to manage SharePoint from the server point of view. Here you need hearty skills in MS SQL, IIS, and, well, SharePoint of all flavors. There's usually a migration involved, and you name it, they range from WSS 2.0 and Portal Server migrations to the state of the art MOSS 2007.

Architects need to know how to cluster MS SQL boxes, distribute WFE servers, and to setup search indexing and queries on a massive scale.

"SharePoint Specialist"

My momma told me I was special, yet it turns out I was a specialist. The specialist is a trenches kind of person. They live with the SharePoint end-user and guide them to the proper care and feeding of a SharePoint site. This is a thankless job. You have users that hate SharePoint with a passion, yet you need to train them how to use it effectively.


Often times, companies are looking for that:

"SharePoint Einstein"

I missed the point with ECM in SharePoint

I've read in the recent blogosphere where SharePoint is utilized mainly as a replacement for a file server. Of course, SharePoint is so many things... It stores your documents, it makes a sandwich. I think it's okay for it to start as a file server replacement, a modest start to building a true Enterprise Content Management system... Other SharePoint features should grab hold over time.

I'm the key SharePoint huckster in my academic organization. I've been selling the OOB Document Library as if it were the second coming. A list posted on the bulletin board behind my desk reminds me of the doclib advantages. I'll go through what I have, dismiss each point, and note a huge advantage I forgot below:

Versioning: You can almost get the same thing by enabling "Track Changes" in Word. Also, our backup plan has "versions" of docs for the last 2 weeks.

Document Check-out: Over the file server, you can't open a file that's locked for editing. Do you really want to know who has the doc open on their desktop? It's probably Johnson...

Meta-Tagging: I have the least to say about this, because, honestly, I don't understand it. My best guess it that it's available search metrics for a file... Stuff about a file in the columns of a doclib.

Alerts: Some love 'em, some hate them... In an organization that is communicative and transparent, you're going to care less when a new version of a doc has changed or docs added...

Workflows: Yes! Another way to be ignored by your superiors, especially with the OOB Approval flows. I have yet to see collect feedback work... People just filter email from SharePoint into the trash.

Search: I love SharePoint search, and its spidering process rocks out... Why have multiple desktop search processes running, if everything is stored in SharePoint? Of course, not everything is stored in SharePoint and the Search does not spider workstation drives, where most content lurks, waiting for a hard drive failure.


Okay, now the single greatest tool within SharePoint, that makes it an Enterprise Content Management system vs. just a file server replacement. The tool I know little about and promise to learn everything there is to know about from this day forward as long as I live.

*drum roll*

Information Management Policies, specifically, EXPIRATION.

I look on my workstation, my third over 4 years in my current employer. I look at My Docs, and I can see a time capsule of data files going back to day one. All of it is, without doubt, worthless... Since my workstation's hard disk space is cheap, I keep this stuff around. Who knows, I might need that doc that explains how to move mail from Eudora to an Exchange account.

Expiration: "Automatic scheduling of content for processing, and expiry of content that has reached its due date." Published by Microsoft Availability: Available

I can't wait to expire something!

Data Source Library 101

Okay, boffo stuff going on here, really. If you're a real .NET developer, have taken a Ted Pattison course, or if you generally have a clue, maybe this is really basic information is not new to you. Here it is... Hold on everybody...

The lists (and libraries) you see within the SharePoint Designer "Data Source Library" are based on automatic queries that SPD creates to display the lists in the first place. If you want to add your own queries to the data in these lists, you must use the "Copy and Modify" command within SPD.

"Copy and Modify" solved my problem of not being able to select limited fields and to set the lists property's "Items and folder scope" to "RecursiveAll" so I can see all the files within folders. (Despite my 'Best Practice' advisement, people are still creating folders in document libraries. Don't get me started...)

This only took me about 8 hours to figure out. That's 480 minutes of lost Desktop Tower Defense productivity.