CODECUBE VENTURES

Google API Rocks

I had to do a little market research for someone, and the Google web api came to the rescue.

Paypal wanted the duplicate stores filtered in this query so they could see how many yahoo stores used them. Running the query states there are about 40,000 results so this was obviously not going to be a manual job. To solve the issue, I made a .NET console application that queries the google web service, and saves the results in an xml file. The reason I had to go that route was that Google only returns 10 results at a time, and I didn't want to tie up that many SQL Server resources while the querying ran.

So I set it to run overnight, and when I came back, there where 990 results in the xml file ... I was flabbergasted. After a bit of poking around, it turns out that even though Google estimates 40,000 results, it will only return 1000 results for any given query ... this is true even in the web interface of google.

So I had to work with what I had ... I then wrote a small app to import the xml file (which was simply a DataSet persisted as XML) into a Database table from which I could then issue SELECT DISTINCT queries to get the answer I was looking for. So that was a cool experience of working with .NET, Web Services, and ADO.NET ...

Latest post: Digging Up the First Version of CodeCube

See more in the archives