Infreemation

Η αναζήτηση παίρνει εξυπνώτερη, παίρνουμε stupider

Πολύς έχει γραφτεί πρόσφατα στον τρόπο με τον οποίο η ευφυής αναζήτηση θα λύσει όλα τα είδη προβλημάτων, πρόσφατα μέσα Το τέλος της θεωρίας, Chris Άντερσον από η «μακριά ουρά» τη φήμη συγχέει την αφθονία χαμηλών κρεμώντας φρούτων σπου η «μεγάλη αναζήτηση» και οι βιοτεχνολογίες παρέχουν τη δυνατότητα να γίνει κατανοητή πραγματικά και να εξαγάγει η σημασία, να θέσει και να πλαστογραφήσει ή να υποστηρίξουν hypothesies. Το Mathew Ingram παίρνει το ζήτημα με το συνδεμένο με καλώδιο άρθρο μέσα Google και το τέλος όλων και σωροί του Alistair Croll επάνω μέσα Η μεγάλη αναζήτηση αλλάζει την επιστήμη; υπογράμμιση της γνωστής επιστημονικής επωδού: ο συσχετισμός δεν απαιτεί την αιτιολογία.

Για να είναι δίκαιο στο Chris, φαίνεται ότι καταλαβαίνει το σημείο Mathew ότι ο συσχετισμός δεν είναι αιτιολογία, μάλλον η διατριβή του φαίνεται να είναι αυτή με τα αρκετά μεγάλα σύνολα δεδομένων και τους ισχυρούς υπολογιστικούς αλγορίθμους, αιτιολογία προσεγγίσεων συσχετισμού. Εντούτοις πλαισιώνω με Mathew και Alistair, δεν σκέφτομαι ότι ο Chris καταλαβαίνει ποια Google ή γρήγορη αλληλοuχία γονιδίων φέρνει στην επιστημονική ανάλυση, ή έχει γράψει ένα άριστο σατυρικό άρθρο:

Το Petabytes επιτρέπει σε μας για να πει: «Ο συσχετισμός είναι αρκετός.» Μπορούμε να σταματήσουμε τα πρότυπα. Μπορούμε να αναλύσουμε τα στοιχεία χωρίς υποθέσεις για ποιος δύναμη παρουσιάζουν. Μπορούμε να ρίξουμε τους αριθμούς στις μεγαλύτερες συστάδες υπολογισμού που ο κόσμος έχει δει πάντα και άφησε τους στατιστικούς αλγορίθμους να βρούν τα σχέδια όπου η επιστήμη δεν μπορεί.

Ηχεί όπως εμείς πρέπει να είναι σε θέση να καθίσει ακριβώς πίσω και να τροφοδοτήσει τα ακατέργαστα στοιχεία έναν ογκώδη υπολογιστή σύννεφων, να αρπάξει μερικούς καφέδες, να ζήσει μερική διάρκεια ζωής και να πάρει μερικές απαντήσεις (Βαθιά σκέψη καθένας;). Δεδομένου ότι η τεχνολογία αναζήτησης παίρνει εξυπνώτερη μπορούμε όλοι να αντέξουμε οικονομικά να πάρουμε πολλή stupider, όπως δεν πρέπει πλέον για να λύσουμε τα επιστημονικά προβλήματα.

Στην επικαιρότητα ο αλγόριθμος Google pagerank και Craig Venter το κυνηγετικό όπλο DNA που τοποθετεί διαδοχικά τις τεχνικές είναι επιτυχείς επειδή είναι υπερβολικά απλοϊκοί, σχεδιασμένος για να συλλάβουν τα χαμηλά κρεμώντας φρούτα όσο το δυνατόν γρηγορότερα, δεν λύνουν τα σκληρά προβλήματα - μάλλον μας παίρνουν γρηγορότερα κάτω από έναν δρόμο που οδηγεί σε περισσότερες ερωτήσεις. Οι ερωτήσεις που είναι πιθανές που περιπλέκονται πάρα πολύ για είτε τις μηχανές αναζήτησης είτε τη χαριτωμένη βιοτεχνολογία εξαπατούν για να απαντήσουν. Απαιτώντας τα πειράματα και τις αναλύσεις που είναι πάρα πολύ περίπλοκα και λάθος-ευαίσθητα… ότι ανάγκη να είναι φορητά, πειμένα και καλοπιαμένα. Η επιστήμη στον πραγματικό κόσμο είναι τόσο διαφορετική από το πλατωνικό πρότυπο που διδάσκει στα σχολικά βιβλία. Η αποτυχία είναι σημαντική, τα λάθη είναι κρίσιμα και προχωρούμε επειδή η ανθρώπινη σκέψη είναι εντυπωσιακά προσαρμόσιμη και ελαστική παρά αυτό. Αντιπαραβάλτε αυτό στους τύπους προβλημάτων που θα πάρουμε όταν καθοδηγείται η ανάλυσή μας από τους κατάμεστους λαθών αλγορίθμους υπολογιστών, που μολύνονται με τα σκουλήκια, και το στοιχείο διατρυπείται με τα λάθη και spam.

Μέχρι τη δύναμη υπολογισμού και τους αλγορίθμους που το καθοδηγούν, αληθινά evolutionarily σχεδιάζεται, δεν σκέφτομαι ότι η επιστήμη θα μάθει πολλής από τον υπολογιστή. When we do get the kind of AI that Chris and the Google founders are looking for, I suspect that they will find it impossible to clock that type of artificial intelligence at Gigahertz speeds, and that we may end up re-evolving a computer that looks and acts very similar to the human brain. At which point we may regret not using the ones we already have instead.

For the next stop on this train of thought, read the excellent article Is Google Making us Stupid? I’ve got one foot in the YES camp.

Addendum: the Wired article bothered me as an epitome of reductionist scientific thought. Reductionism by nature tends to focus on the simple problems, hard problems which are complex and expensive to tackle are avoided which leads to the amplification of reductionist techniques and causes. Sooner or later you might be convinced that all knowledge is within the reach of such reductionist approaches. There is a disturbing correlated trend for industry funding of scientific research to further skew science by leaving problems without obvious economic payoffs by the wayside. I would suggest that both industrial and reductionist science are represented in the Wired hypothesis.

Posted in news | No Comments »

Cloud computing - linear utility or complex ecosystem?

Reuven of Enomaly speculates on whether there will be an analogue of Moore’s law for cloud computing, looking to coin “Ruv’s law.” I would like to see more detail on what it would postulate, presumably a linear relationship between growth in cloud computation and time. I think we would also agree this would need to stand the test of time before it would be considered “law.” Moore referred to a rather simple relationship between the number of transistors that can economically be used in electronic chips and time. The cloud is likely to become a very complex ecosystem, and defy simple linear rules of productivity. Rather I would expect the cloud to both behave in unexpected ways and exhibit emergent properties. On that note I am much more interested in the phase transitions, critical junctures where the properties of the system change radically, and what the underlying causes might be (technological breakthroughs, human behaviour, power shortages). I wouldn’t be shocked if the behaviour of clouds was as hard to predict as the weather (”5 day forecast calls for a 200 msec second standard deviation in latency with 10% probability of the jitters”) or the stock markets. I’m only slightly joking - my early experiences with sharing hosted grid computing resources have been variable (Mediatemple and Mosso have low cost plans). In any case I look forward to more clarity on cloud structure, composition, performance, any potential “laws” and above all the likelihood of rain… Anyone interested in a lively string of Q&A surrounding the much hyped “cloud computing” revolution should look in on the Google group for cloud computing and check what the insightful Alistair Croll of Bitcurrent has to say. Lots of folks are trying to define cloud computing these days (check out defogging the cloud for a nice simple explanation), and its hard to do partly due to a Cambrian explosion of diversity which makes the cloud(s) a fast moving target. As for me, I’m embracing the trend from the web operations trenches while keeping my sense of humour about the hype:

Posted in news | 2 Comments »

Awesome Magnetic Visualization

Semiconductor’s Magnetic Movie is a stunning, if questionably accurate visualization of magnetic fields and their interactions. Worth a watch:

Magnetic Movie from Semiconductor on Vimeo.

Posted in news | No Comments »

Goaltender, Bitcurrent, and Miscelania

Finally got www.goalr.net into the Heroku private beta which will allow the private domain and e-mail functionality that our “Goaltender” application relies upon to both gather our weekly goals and followup on them.

Just posted to Bitcurrent on the future of cloud computing.

Business has been moderately insane (in a good way) and we’ll be moving offices downtown shortly. Somewhere in the last few weeks a house was purchased, Alio blogged about it on our new blog “now we are three“.

Posted in news | No Comments »

Blitzweekend project: getting real with GoalR

Ever since I attended 37 signals “Getting Real” workshop I’ve want to put some of their design principles to the test. My company is Syntenic, a boutique consulting and services shop primarily focused on web operations, performance optimization via application delivery controllers (load balancing and server offload) and 24×7 high availability geographically distributed architecture. What this means is that we are usually helping enhance the performance and reliability of our customers applications instead of building our own. Recently this has been changing as we increasingly are redesigning or developing applications from scratch for our customers.

CodeBlitz
Blitzweekend in Montreal has given us the opportunity to give the getting real approach and Ruby on Rails in particular a whirl. Our challenge was to come up with a project idea that was feasible in one weekend including conception and coding. It didn’t take much to convince our web codemonkey extraordinaire Will Stevens to take on the challenge. Lucky for us since he has to do all the real work (including learning rails on the fly)! Alistair Croll joined on as our resident marketing genius. The onus was on me to come up with a simple enough project, which ultimately was inspired by an article I read several years ago about one of Google’s management approaches: employees were asked to list their objectives at the start of every week and report at the end of the week which of their objectives have been obtained. I’m not a fan of task management, listing and tracking to-dos can take more time than getting them done. The big challenge I have from a management perspective is keeping all my objectives or in my sights, the tasks required to complete the higher level goals seem to follow easily as long as I can keep focused on the goal. You don’t become a great goal scorer by looking at the puck, thinking about stick handling or ball dribbling. Great players keep their head up and keep their eyes on where the puck or the ball needs to go.

Aren’t you a little short for a goaltender?
Perhaps the sports analogy isn’t perfect but I’m going to run with it. The application is codenamed “goaltender” shortened to GoalR (www.goalr.net) because all the other domains were taken. GoalR e-mails you at the start of the week asking you what your objectives for week are, the goals in question. At the end of the week you are prompted to indicate which goals you managed to “score,” or worse yet admit defeat. We are looking to allow the user to score. If we have time we will introduce the concept of teams and rosters, allow team members to track, encourage even heckle their fellows in pursuit of their goals. We’d love to introduce the concept of assists, if we can figure out how that would work.

First goal of the game…
Alistair’s here, back to work (an no doubt 100 digressions a minute). Our objective, “goal” if you will, is to build a working goal management application (not another task manager!). It needs to be functional and useful by the end of this weekend, despite attending Steph’s 30th birthday party tonight. We’re keeping it real and trying to make management fun. Might be an unrealistic goal but hey, if you don’t take shots you’ll never score a goal.

Posted in Uncategorized | No Comments »

Algorithmic efficiency in the face of inelegance

Seed magazine published a wonderful crossover article between the worlds of biology and engineering “Algorithmic Inelegance”

The complexity of developmental regulation isn’t a product of design at all, and it’s the antithesis of what human designers would consider good planning or an elegant algorithm. It is, however, exactly what you’d expect as the result of cobbling together fortuitous accidents, stringing together helpful scraps into an outcome that may not be pretty, but it works. That’s all evolution needs from developmental processes: something that works well enough, no matter how awkward or needlessly complex it may seem.

Biological solutions can be remarkably efficient (well adapted thanks to evolution by natural selection) while being absurdly backwards in design. Nerves in the mammalian eye pass on top of the rods and cones that collect light yet I can’t complain about my eyesight (other than the need for contacts but that’s another story). Cephalopods apparently have more accurate vision thanks to happier circumstances:

The vertebrate retina is wired “backwards”. That is the photoreceptors point to back of the retina, away from incoming light, and the nerves and blood vessels are on the side of the incoming light, this means that any image formed on the vertebrate retina has to pass though layers of blood vessels and ganglion cells, absorbing and distorting the image….Now consider the eye of squids, cuttlefish and octopi. Their retinas are “rightway round”, that is the photoreceptors face the light, and the wiring and the blood vessels facing the back (1). Squid and octopi have no blind spot; they can also have high visual acuity. The octopus also has a fovea-equivalent structure, which it makes by packing more (or longer) photoreceptors into a given area (1). Because it doesn’t have to create a hole in the supporting tissue it can have arbitrarily large “fovea”, and greater visual acuity. Cuttlefish have better visual acuity than cats (2) and because of their “rightway round” retinas; this level of acuity covers nearly the entire retina (1,2) unlike vertebrates where it is confined to the small spot of the fovea.

More on this fascinating story here.

The author seems to indicate that human engineers would shy away from the needless complexity that evolution by historical accident seems to create. However, today’s leading engineers faced with shifting requirements, overwhelming problem complexity, or incomplete understanding of the science behind the solution may well adopt an evolutionary strategy toward solving a problem, as many circuit and drug designers have already done successfully. (I’ll try and dig up some references). They start with a modular design with lots of simple parts that can be tweaked and rearranged quickly, set up replication with random modification and a process for selecting the best fitted solutions. Like artificial selection, or breeding, you can achieve results quickly. This is similar to what successful organisms do, molded by the evolutionary process of natural selection working hand in hand with mutation and reproduction. And it explains why we burn brightly on this earth, have sex, and above all why we are programmed to die.

Posted in Uncategorized | No Comments »

The legal limits of creativity explored: Recut Reframe Recycle

Danah Boyd points out that the center for social media has released an informative paper entitled Recut Reframe Recycle concerning the boundaries of fair use (fair dealing in Canada) in digital media.

Types of use they point out:
• Parody and satire
• Negative or critical commentary
• Positive commentary
• Quoting to trigger discussion
• Illustration or example
• Incidental use
• Personal reportage or diaries
• Archiving of vulnerable or revealing materials
• Pastiche or collage

Some of these can push the limits of fair use, take for example D.J. Danger Mouse’s Grey Album (a mix of the Beatle’s White Album and Jay-Z’s The Black Album) which I discussed in Mashups: from hobby to art form to controversy.

The report concludes:

Some of these practices also fit comfortably into the evolving pattern of fair use jurisprudence. By contrast, other video makers appropriate material wholesale and without context or comment, in ways that clearly are not fair use. In all these cases, informed judgment on fair use, following established precedent, should be relatively straightforward. Many times, however, for instance within the category that our researchers called “pastiche or collage,” creators are developing practices that are at or near the boundaries of contemporary fair use analysis. Traditional fair use analysis would neither definitively exclude nor include them—at least until there is a better understanding of motive, context, circulation, and use of the new works. Since fair use doctrine evolves with creative practice, these borderline cases provide important areas for future research and analysis.

They call for “a code of best practices around fair use in online video needs to be articulated, both to educate new makers and to provide guidance for regulators private and public.” It will be a contentious process to push for more definition in the massive legal grey areas that exist in copyright law regarding fair dealing but it has such big implications for the (legal) limits of creativity that creators everywhere should be pushing hard for clarity. Unfortunately the effect of the law thus far has been largely the reverse: Danah gives a good background on why this is such a critical issue in the USA currently:

fair use is quite tricky because courts address it on a case by case basis after someone is sued. There is no list of what constitutes fair use. Thus, remixers engaging in practices that would collectively be viewed as fair use never have certainty that what they’re doing is legal. Because court cases are extremely costly (especially for the lone defendant in the face of Big Mega Corp), corporations can wield a lot of power through the egregious use of “Cease and Desist” letters. Most creators bow down in the face of them even if what they’re doing is totally legit because they are terrified of being sued. In legal terms, a “chilling effect” is when practices are squelched by fear of persecution. Right now, when it comes to remix, we’re in the middle of an ice age. The Chilling Effects Clearinghouse website attempts to counteract some of this effect by collecting and publishing Cease and Desists and other nefarious attempts by corporations to silence fans and critics.

This problem is increasingly relevant to Canadians as a US-corporation sympathetic government in Canada enables and encourages such business practices.

Posted in Uncategorized | No Comments »

Android - one step closer to freedom of communication

Like most heavy cellphone users I have a love hate relationship with both the device and the wireless service. Love because it gives me freedom from a physical location and thus is an essential business enabler for entrepreneurs. Hate because it restricts my freedom in many ways that are designed solely to make more money for the handset and communications provider:

So thanks to Google and others for supporting the development of an “open” platform for phones which threatens to transform the market:

Next step: we need openly accessible wireless networks for such phones. Currently the only really open network is the internet and only if you can afford an unfiltered and reasonably symmetrical last mile which rules out many consumer internet products including wireless data (which is cost prohibitive in Canada). Widespread internet access is only somewhat accessible if you are a hotspot hacker extraordinaire.

Maybe, just maybe Google’s plan to bid for the former UHF 700Mhz spectrum and make it accessible to 3rd parties is connected with Android project. Just maybe ;)

To be fair, Android is getting a lot of hype almost a year before we can expect to see anything, while there are other mobile phone operating systems vying for a place as relatively open mobile internet platforms. Om Malick just published a review of the market for mobile platforms pointing out alternative mobile Linux platforms, and recently Alistair introduced us to Nokia’s N810 tablet which shows potential as an open mobile communications platform even if it doesn’t have an integrated cellphone. It will be interesting to see the next platform moves of Microsoft, currently with a relatively open platform as mobile platforms go, and RIM, whose tightly restricted Blackberry platform is a favorite with enterprise and e-mail junkies.

Posted in news | No Comments »

Can the web help fight corruption?

Lessig turns his keen insight and trademark presentation style towards the analysis of corruption as he steps away from 10 years of focusing on intellectual property issues and their impacts on culture, creativity and innovation. I wish him luck as he will have the full force of the political and business establishments fighting him to their eventual unlamented demise.

For a quick take on the hour long lecture, skip to 8:23, a story about how the sugar institute managed to influence the food nutrition board (FDA) to set the recommended maximum intake of sugar to 25% of your daily calories, instead of the WHO’s advised maximum of 10%. Hang on through 10:30 where he talks about how lobbyists have skewed the global warming “debate.” Other issues he covers are the pharmaceutical industry targeting doctors with “bribes that are not considered bribes” and the influence of private interest funding on “scientific” research. His bottom line is that everywhere we look, money determines decisions more than ever, often trumping what is in the interests of the citizens.

It all seems depressing, but what gives Lessig and the rest of us hope for democracy is the potential of the internet, particularly the web, to render the forces behind policy decisions more transparent and available to a wider population of stakeholders, namely the citizens (I know this is a shocking concept!). Perhaps the availability of hitherto inaccessible information will inspire more of the population to become politically involved. I’m not yet holding my breath on this one, but let’s dare to dream.

Towards the end of the lecture, Lessig highlights MapLight which exists to expose the relationship between legislators voting patterns and their sources of funding in US congress. Also featured is the Sunlight Foundation:

The mission of the Sunlight Foundation is to use the transformative power of the Internet and new information technology to enable citizens to learn more about what Congress and their elected representatives are doing, and thus help reduce corruption, ensure greater transparence and accountability by government, and foster public trust in the vital institutions of democracy.

Aside: Canadians should check out the Data Libre project.

Corruption is a problem that is probably a human condition arising from social interaction and economics, unlikely to disappear, but perhaps like many of our weaknesses, it can be minimized. We are in the earliest days of the development of new tools that combine openly accessible data, visualization technology, and the reach of the web. It remains to be seen if they will help improve our democracy or become yet another system to be influenced and gamed by special interests.

Posted in Uncategorized | No Comments »

The Web as operating system

Alistair, who has been writing recently at GigaOM, puts together the most lucid synthesis I have seen on the topic of the “Web OS.”

Big Internet companies are making themselves the OS of the web 2.0 world. In addition to the fundamentals — operating a web application, storing data, handling logins — each company has a core expertise. In Google’s case, it’s page ranking and relevance; Facebook maps social relationships; Salesforce knows about customer relationships; and eBay has an auction and reputation engine.

Each of these web OS service platforms by their natures can in theory interoperate, and competitors can in theory be mixed and matched. Alistair points out that developers who use these sites as platforms still have tenuous relationships with the terms of use (such as the vibrant Facebook developer community). This will be pretty contentious, as many of the big web site “platforms” have incentives to build their own apps. But all software platform companies realize that developer communities are the key to success, and if Facebook doesn’t find the right balance of developer rights and incentives someone else will. In an ideal world we assume a free market where the user can basically choose their web OS options from a range of competitors. The question is how much competition can the market sustain (can there be multiple platforms for each service (who competes with eBay?) and can there be multiple winners within the ecosystems that run “atop” those platforms. I think the answer to the latter is yes, but my concern is with the former (competition for platforms) and how much collusion there will be between the established “components” to band together monopolistically (eBay and Facebook sitting in a tree), which has been a bit of an issue in the software world.

In a slight digression I leave you with the following totally awesome video on the nature of information and how the web is setting it free:

Posted in Uncategorized | 1 Comment »

Can code be bad for the planet?

Alistair has an interesting post on Earth2Tech, the thesis of which is that inefficient coding practices can lead to environmental harm.

I almost misled myself into thinking AC is blaming virtualization and SaaS/IaaS (infrastructure as a service) on creating such inefficiencies. Rather, he skipped past the obvious environmental benefits of server consolidation (improved resource utilization) and service centralization (via economies of scale) and instead builds on the consequences: code inefficiencies become more obvious when you’re no longer massively overprovisioning hardware. This is an opportunity as much as a challenge. Bad code did matter previously, but not to the extent that is forseen: we’re scaling web applications to a much larger degree than ever before. There are more users, and inefficiencies are multiplied.

So it is great news that virtualization and on demand infrastructure will allow us to focus more on code efficiency since as Alistair (incidentally a veteran in the monitoring of applications) points out, it exposes more granular economics of computing. These technologies are paving the way to greater infrastructure efficiencies and by forcing better utilization of hardware, putting more focus on the efficiency of the code that cohabits the infrastructure.

Increasing code efficiency has been generally unimportant except in edge cases. Stability and function has been more of a concern while Moore’s law and incomplete costing of infrastructure have been more than compensating for performance. Rapid application development platforms proliferate based upon the abilities of modern hardware to crunch “affordably” through the multiple layers of abstraction.

What me worry?

There’s definite potential for code to have an environmental impact. We have an existing ecological disaster on our hands with the castoff personal computing hardware of both enterprises and consumers. Almost all of that computing power was wasted idling, never used, just for the ability to load Microsoft Office applications quickly. How do we acheive more efficient code? As more people rely more on computing, the costing of which is becoming more accurate and granular, and as the barriers of entry for developers drop, we should witness an evolutionary process at work battling inefficiency assuming:

• large population of users
• competing applications
• rapid generation spans with modification
• market exerts selective pressure

While I think these evolutionary forces are already at work, the selective pressures have been weak, the environment has been overly abundant leading to a Cambrian explosion of inefficiencies that eventually will be represented in costs that the market will react to, assuming that the market has the freedom to do so. This is where intellectual property issues and the “one platform to rule them all” attitude may present a bit of a speedbump, but only that.

Alistair’s most important point is highlighting that proper costing of computing is essential: if we want to minimize environmental impact we need to measure the efficiency of work performed by applications and the true cost of the resources they consume. My conjecture is that an evolutionary process of anthropogenic artificial selection, automated or not, should optmize resource utilization. This rests upon the premise of a competitive market, which I believe we are just starting to see in the world of software.

For now I’m much more concerned with how poorly conceived code can compromise privacy, and restrictive code that restricts our freedom of communication and innovation. But those are stories for another post.

Posted in Uncategorized | No Comments »

The world’s on fire

The funds that Sarah McLachlan would have spent to produce a music video are donated for more practical uses, and the video is instead used to help us visualize just how much of a difference $150,000 can make to those less fortunate than us:

I’m a long time fan and proud she is a fellow Canadian.

Posted in Uncategorized | 2 Comments »