Comments on: Yottabytes and the Data Analysis Challenge

By: Company - News - Solera Networks™

Company - News - Solera Networks™ — Tue, 24 Nov 2009 18:23:26 +0000

[...] Secrecy works to challenge excessive government secrecy and to promote public oversight”) in a post on the challenges of dealing with large data sets. The December 2008 JASON (not an acronym) report [...]

By: Grumpy Old Man

Grumpy Old Man — Fri, 09 Oct 2009 17:10:23 +0000

You are saying that data including oceans is totally useless. You are talking about the engine of the world and calling it totally useless. I think you might be thinking with blinders on.

By: taochiapet

taochiapet — Tue, 07 Jul 2009 16:09:53 +0000

i think NMvoiceofreason’s point is that we’re already capable of, and indeed do, generate yotta’s worth of image data. the post’s unrealistic use of 1 byte = 1 m^2 (granted, for illustrative purposes) tends to obscure that fact.

rather than an increase in the rate that imaging satellites produce data, perhaps there’s just an increate in the use of imaging satellites to collect data. in that regard, i’d say the only memory being used up is the collective memory of a government limited in power and constrained by law…

By: Matthew Tanner

Matthew Tanner — Tue, 07 Jul 2009 13:32:55 +0000

NMvoiceofreason, (July 7th, 2009 at 8:32 am)

Your twinky analogy is missing something. I get the resolution we’re working at but that says nothing about the rate that we’re using up memory. I’m presuming that it’s has something to do with the rate imaging satellites produce data.

By: NMvoiceofreason

NMvoiceofreason — Tue, 07 Jul 2009 12:32:17 +0000

If you image the entire earth at 128×128 per meter resolution (roughly 3″ squares), then you reach a yottabyte in about a month. 2/3rds of that data is almost totally useless, as it shows the surface of the ocean. If you go multispectral, C/m/y/k/Ir/Uv you can use a yotta per week. If you add lidar and sar, you can do a yotta a day.

Nothing is unmanageable with sufficient time, money, manpower, except for the bureaucracies that gather sensor data.