2018/08/23

Scheduling Notebooks at Netflix – Netflix Technology Blog – Medium

Scheduling Notebooks at Netflix – Netflix Technology Blog – Medium:

Notebooks are, in essence, managed JSON documents with a simple interface to execute code within. They’re good at expressing iterative units of work via cells, which facilitate reporting and execution isolation with ease. Plus, with different kernels, notebooks can support a wide range of languages and execution patterns. These attributes mean that we can expose any arbitrary level of complexity for advanced users while presenting a more easily followed narrative for consumers — all within a single document.
Since Papermill doesn’t modify the source notebook, we get a functional property added to our definition of work — something which is normally missing in the notebook space. Our inputs, a notebook JSON document and our input parameters, are treated as immutable records for execution that produce an immutable output document.

2018/07/16

We just watched a U.S. president acting on behalf of a hostile power - The Washington Post

Max Boot lays it out.

We just watched a U.S. president acting on behalf of a hostile power - The Washington Post: "We are past the point when such statements can be dismissed as naivete — especially when Trump has previously admitted that Russia did hack the election. We are past the point where Trump’s conduct can be ascribed to his belief that it is imperative to improve U.S.-Russia relations. If Trump doesn’t care about the state of U.S.-German relations, U.S.-Canadian relations or U.S.-U.K. relations — all of which he has damaged in the past month — why would he care about U.S. relations with a country that has one-fourteenth of America’s GDP and one-tenth of its defense budget? We are past the point where Trump’s conduct can be ascribed to his general sympathy for dictators. If he is waging a trade war on China’s dictator, why is he cozying up to Russia’s dictator? We are past the point when Trump’s conduct — which leaves future elections wide open to Russian manipulation — can be ascribed to his unwillingness to do anything that will tarnish his glorious victory. It is true but insufficient to point out that that Trump’s unwillingness to acknowledge the Russian attack on America is putting his own interests above the country’s. Even if Trump were thinking only in terms of his own political survival — his usual mode — he would be tougher on Putin, because he must realize that kowtowing to the Russian only strengthens suspicions of collusion. But Trump just cannot bring himself to do it. Is that because he hopes for more aid from Putin in the future — or because he is afraid of what Putin can reveal about him? Either way, he gives every impression of betraying his oath of office. U.S. Constitution, Article III, Section 3: ‘Treason against the United States, shall consist only in levying war against them, or in adhering to their enemies, giving them aid and comfort.’ Trump’s own national security adviser said the Russian election attack constituted an ‘act of war.’ So what does that make his boss? Some — including former CIA director John Brennan — now dare call it treason. That conclusion was once unthinkable. No longer."

2018/07/09

Extreme Heat Event in Northern Siberia and the coastal Arctic Ocean This Week – Ocean's Wrath

Extreme Heat Event in Northern Siberia and the coastal Arctic Ocean This Week – Ocean's Wrath:

An extreme heat event for this particular region…with high temperatures of greater than 40 degrees F (greater than 20 C) above recent normals…will impact the coast of the Arctic Ocean (specifically the Laptev Sea and Eastern Siberian Sea) Wednesday-Friday. This will generate maximum daily temperatures as high as 90-95 degrees (32-35 C) near the open ocean coast! Yes,  you read that correctly.
Meanwhile we continue extracting and leaking methane into the atmosphere. We continue generating and releasing CO2 at a high rate. Here in the United States, when we could be choosing representatives who have even slight interest in this problem, we are busy voting for the end of democracy. I marvel at our stupidity.

2018/04/18

"Be a Good Ancestor"

China made solar panels cheap. Now it’s doing the same for electric buses. - Vox:

The rapidly growing megacity of Shenzhen, China, was choked with diesel pollution in the early 2010s. Though buses were just 0.5 percent of the city’s vehicles, they were responsible for 20 percent of the air pollution. So, as the World Resources Institute recounts, Shenzhen electrified them. All 16,359 of them.
It got there through some cleverness — they planned out charging infrastructure and pushed manufacturers to offer lifetime warranties on batteries — but mainly through government subsidies, which covered up to half the upfront cost of the vehicles.
In 2013, China was bopping along with 1,700 BEBs. ...it realized that its rapidly expanding urbanization could not continue to rely on diesel buses, lest it make an already crippling air-pollution problem even worse.
So it decided to make electric buses a thing. How? By dumping a giant pile of money on the problem, subsidizing the purchase of more than 350,000 BEBs in the following four years.
Other cities are beginning to get on the BEB bandwagon as well.
[BNEF writes that] “Paris aims to electrify all of its 4,500 buses by 2025, Copenhagen has committed to procure only zero-emission buses from 2019, and Los Angeles has the same target for its fleet of 2,200 buses by 2030.”
sometimes problems are big and urgent, you need scaled-up solutions quickly, and you just don’t have time to mess around. China didn’t nudge its solar industry, it kicked it in the pants. Now it’s doing the same to the BEB industry.
As economists will rush to point out, there have been enormous inefficiencies and waste along the way, money lost on graft or junk technology. Solar has often been poorly installed or curtailed. According to BNEF, some 43 percent of the BEBs allegedly produced in China in 2015 were basically fraudulent.
[China] rightly recognized that the problems (coal and diesel pollution) are intensely urgent and it’s not that huge a mystery how to solve them.
Amortized over the next 50 to 100 years, the next several generations of humanity, the cost of transition are a screaming deal at almost any price. It’s just a matter of learning (or perhaps relearning) to take a long-term view of human interests — to be a “good ancestor,” as they say.

2018/04/15

Installing Ubuntu 17.10 on a 2013 13" MacBook Air: Wifi

Today I am remembering why I decided to give Mac OS X a try back in 2002: I got tired of spending most of my time configuring linux drivers. I thought the situation might have improved in the interim, and it probably has. Unfortunately, the universe of extant hardware is huge, and it's hard to fill all of the holes. Anyway, today I'm trying to replace macOS with Ubuntu 17.10 on my 2013 13" MacBook Air. Most of the installation has gone smoothly, with the main exception being Wifi. After more than an hour of flailing I found this, and it appears to have worked:

BCM43142 WiFi driver in Ubuntu. | How to install BCM43142 WiFi Driver in Linux

I had to copy a lot of packages from the Ubuntu installer thumb drive, not just the two mentioned in the article. The reason was that the Ubuntu installer didn't install gcc 7 or its supporting packages. Here are the packages that I needed to copy from the installer thumb drive. All files were located under .../pool/main:
  • bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu3_amd64.deb
  • dkms_2.3-3ubuntu3_all.deb
  • gcc_7.2.0-1ubuntu1_amd64.deb
  • gcc-7_7.2.0-8ubuntu3_amd64.deb
  • libasan4_7.2.0-8ubuntu3_amd64.deb
  • libatomic1_7.2.0-8ubuntu3_amd64.deb
  • libc6-dev_2.26-0ubuntu2_amd64.deb
  • libc-dev-bin_2.26-0ubuntu2_amd64.deb
  • libcilkrts5_7.2.0-8ubuntu3_amd64.deb
  • libgcc-7-dev_7.2.0-8ubuntu3_amd64.deb
  • libitm1_7.2.0-8ubuntu3_amd64.deb
  • liblsan0_7.2.0-8ubuntu3_amd64.deb
  • libmpx2_7.2.0-8ubuntu3_amd64.deb
  • libquadmath0_7.2.0-8ubuntu3_amd64.deb
  • libtsan0_7.2.0-8ubuntu3_amd64.deb
  • libubsan0_7.2.0-8ubuntu3_amd64.deb
  • linux-libc-dev_4.13.0-16.19_amd64.deb
  • make_4.1-9.1_amd64.deb

2018/01/21

Trump’s invincible ignorance

Last Week Was a Degrading Embarrassment. And a Microcosm of Trump. – Foreign Policy: Does Donald Trump engage in some secret ritual involving the mass ingestion of bullshit? How else does he manage to spew out so much?

Almost everything else about that tweet is also false: The decision to sell the old U.S. Embassy was made by former President George W. Bush, not former President Barack Obama, and it was done for security reasons. The new embassy, built at a cost of $1 billion, not $1.2 billion, has been described by Trump’s own ambassador to the Court of St. James’s as a “bargain” that was paid for by the sale of other U.S. properties in London.
...in fact there is a big price to be paid for having a president whom no one can trust. It makes it almost impossible, for example, to convince other countries to go along with the United States in any international crisis where the facts are in dispute.
“See, that’s treason right there,” he said. No it’s not — unless treason is redefined to mean, as it does in places like Turkey and Russia, any criticism of the supreme leader.

2018/01/17

What Makes the Hardest Equations in Physics So Difficult? | Quanta Magazine

What Makes the Hardest Equations in Physics So Difficult? | Quanta Magazine:

When you start modeling a flow using Navier-Stokes, your fluid will have some initial amount of energy. But in a turbulent flow, that energy can get concentrated. Instead of being distributed evenly across the river, kinetic energy may gather in arbitrarily small eddies, and particles in those eddies could (theoretically) be accelerated to infinite velocity.
... turbulence is meant to describe exactly this — the transfer of kinetic energy from large to smaller and smaller scales...

2018/01/16

Why Raspberry Pi isn't vulnerable to Spectre or Meltdown - Raspberry Pi

Eben Upton provides a wonderfully clear explanation of recently-disclosed CPU vulnerabilities: Why Raspberry Pi isn't vulnerable to Spectre or Meltdown - Raspberry Pi. I think this is the clearest writing on CPU architecture details that I've seen, aside from articles in BYTE magazine in the early 1980s. The comment thread is also a fun read, not least for its mention of the Motorola 68000.

2018/01/03

Ptolemy, Kepler and Fourier walk into a used bookstore...

In which I learn a little modern JavaScript and bemoan my lousy memory: Epicycles Ever After

2017/12/21

Sarah Kendzior on Twitter summarizes Sen. Warner's Red line

Sarah Kendzior:

All Dems and GOP must unambiguously say that any effort to shut down Mueller probe would be a gross abuse of power. This is a red line.
Many of the same people attacking Mueller's appointment today praised him months ago.
Our intel committee uncovered numerous, troubling high-level engagements between the Trump campaign and Russian affiliates.
The national security threat we face demands we rise above partisan differences. We should all want to know the truth about what happened in the election, and know it as quickly as possible
POTUS's track record is a source of concern. Most people thought he wouldn't fire Comey and he did. Firing Mueller would cause a constitutional crisis.
I had hoped I'd never have to make these speech, but these are troubling times. We must all speak up -- before it's too late.
Sen Warner's speech on the Mueller probe felt more like a warning to the US public than to Trump: brace yourself for a crisis even worse than our current one, and don't fall for spin

EPA says Superfund Task Force left no records

Blessed are the scum, for they shall envelop the earth. Scott Pruitt's secretive, unaccountable EPA is re-defining what it means to be a public servant. EPA says Superfund Task Force left no records | News | santafenewmexican.com:

The task force in June issued a nearly three dozen-page report containing 42 detailed recommendations, all of which Pruitt immediately adopted. The advocacy group Public Employees for Environmental Responsibility, known as PEER, quickly filed a Freedom of Information Act request seeking a long list of documents related to the development of Pruitt’s plan.
“Task force members were all volunteers from EPA staff with no selection criteria,” Johnny Walker, a Justice Department lawyer representing EPA, wrote to PEER last month. “The Task Force recommendations were intended to provide guidance to EPA staff. As such, there was no plan for public review and comment on the recommendations.”
The task force was led by Albert “Kell” Kelly, whom Pruitt hired at EPA as a senior adviser at an annual salary of $172,100. Kelly was previously the chairman of Tulsa-based SpiritBank, where he worked as an executive for 34 years.
The Associated Press reported in August that Kelly was barred by the Federal Deposit Insurance Corporation from working for any U.S. financial institution after officials determined he violated laws or regulations, leading to a financial loss for his bank.

2017/12/07

U.S. Sen. Martin Heinrich: Why hinder a rapidly growing industry?

Why hinder a rapidly growing industry? | Albuquerque Journal:

Only a little more than 1 percent of the hundreds of thousands of American solar workers manufacture solar panels or modules. Even within solar manufacturing as a whole, approximately 20 times more American workers work for companies like New Mexico’s Unirac that manufacture other equipment used to produce solar power. Those companies have warned that they may have to lay off employees if the United States imposes costly tariffs or quotas on imported panels.

2017/12/03

I’m on Trump’s voter fraud commission. I’m suing it to find out what it’s doing. - The Washington Post

I’m on Trump’s voter fraud commission. I’m suing it to find out what it’s doing. - The Washington Post:

Instead, the commission is cloaking itself in secrecy, completely contrary to federal law. Recommendations for changes in public policy — whether you agree with them or not — ought to come through an open, public discussion where any American can weigh in.
the memo wasn’t written by staff — it was written by individuals who were later named to the commission but who were working outside of government at the time. The letter went out immediately after our first conference call, indicating that Kobach’s data-gathering effort was underway before the commission formed. But no one told members of the commission that; I learned about it from the press.
Strangely, [Kobach's] charges had less to do with how voters in New Hampshire had conducted themselves than with the structure of the state’s election laws, which Kobach apparently dislikes.
The commission was established by executive order under the auspices of the Federal Advisory Commission Act (FACA), which requires notice of our public meetings, disclosure of our work product and the opportunity for public participation. FACA was written precisely so Americans would know what the government is doing and what it is considering, so we could participate in that process.
Without transparency about the commission’s actions, how can you find out if a policy is being developed that may require you to have a Real ID-compliant driver’s license to vote? Or whether you’ll have to prove American citizenship at the polls? How will you know about proposed changes to voter registration deadlines or new restrictions on absentee balloting?
Of course, this is politics. But remember, we as American citizens are supposed to own the process.

2017/11/26

Gutting net neutrality is a death knell for the resistance - The Globe and Mail

Gutting net neutrality is a death knell for the resistance - The Globe and Mail:

Gutting net neutrality is one of the most unpopular proposals the Republicans have made this year, along with Trumpcare and the proposed tax hike for all but the wealthiest Americans. Normally, there would be risk in proposing such widely loathed policies, but the GOP appears unconcerned with public approval – a strong indicator to me that the elections are indeed a fait accompli.
What can we Americans do? Talk about it – while we still can. Call our representatives, organize in our community, and have a plan for what we'll do should these repressive initiatives pass. Over the past year, citizens have had success exerting public pressure on officials and raising consciousness over social issues. The internet was key to these endeavours, which is precisely why the administration wants to eliminate equal access to it. If we, as Americans, want to retain our voice, we must speak up now, or forever, involuntarily, hold our peace.

2017/11/19

Shaping the World | David R. MacIver

Shaping the World | David R. MacIver

This keynote speech from PyCon UK is about software, a little. But it's more about societies and other complex systems, and the consequences that arise from our simplified models of them. It is full of wonderful associations.

It's pretty amazing that Mr. MacIver was able to speak so evocatively, especially after having had his memory wiped by the sight of 800 or so audience members.

I wish the read-write web actually existed. I would love to be able to annotate his transcript directly. Instead I'll quote, and note, here.

Models of Trees

If you’re a computer scientist, or have had an unfortunate developer job interview recently, a tree is probably something like this.
Hello, Google ;)
[A real, biological tree] provides shade, maybe fruit, it has a complex root system. It’s the center of its own little ecosystem, providing shelter and food for birds, insects, and other animals.

"Complex root system" reminds me of what has recently been written about how tree root systems communicate and share nutrients with other trees. It also reminds of the relationship between root systems and fungal meshes (a mushroom is to the whole fungus as the tip of the iceberg is to the submerged mass), which dissolve rock to serve as nutrients to plants in exchange for carbohydrates (?) from the plant.

And of course it reminds me of Aldo Leopold's story of an atom: the "Odyssey" chapter from "A Sand County Almanac", written in a time when it was believed that plants disintegrated rocks solely by driving roots into them like wedges.

Back to the keynote:

The problem that the German scientific foresters ran into is that complex, natural, systems are often robust in ways that simple, optimised systems are not. They’ve evolved over time, with lots of fiddly little details that have occurred locally to adapt to and patch over problems. Much of that illegibility turns out not to be accidental complexity, but instead the adaptation that was required to make the system work at all.
I've spent much of the past year updating a build system that generates Makefiles from Visual Studio solution/project files. This is a lovely description of that system.

Taxes and Their Avoidance

A picture of a building with multiple windows bricked up.
Have you seen these? Do you know why there are these bricked up windows? Well, it’s because of window taxes... So this is where windows taxes come from – take complex, messy, realities of wealth and pick a simple proxy for it... and you end up taxing the number of windows. Of course what happens is that people brick up their windows to save on taxes. And then suffer health problems...

I was surprised that this was an English building. For me, bricked-over windows and doors evoke Russia, Peter the Great, and the war of countermeasures between tax collectors and peasants.

From "Peter the Great", by Robert Massie:
"The only solution [to Peter's constant need for funds...] was to lay still heavier taxes on the burdened nation... the basic tax had been the old household tax, determined by a census... This tax... made for crowded living because, to avoid taxation, as many families and people as possible crowded under one roof."
"But the new low figures also represented the helplessness of the government to overcome the stratagems of both nobility and peasants who were determined to evade taxes... Russian peasant houses were largely made of logs or timbers notched at four corners. Thus, they could be un-notched in a few hours and either removed to the forest or scattered about. The census takers and tax collectors knew the trick, but there was little they could do about it."
From "For Good and Evil: The Impact of Taxes on the Course of Civilization", by Charles Adams:
"Duplexes and triplexes were built to take advantage of the one-dwelling, one-tax loophole. The tsar's tax collector responded by considering each outside door as a separate household. Peasants responded by boarding up one of the doors..."
But to return to the keynote:
These complex shared ownerships are illegible, so we force people to conform instead to the legible idea of single people or families owning each piece of land. This is where a lot of modern notions of ownership come from by the way – the state created them so they could collect more tax.
...
And of course we have the soviet union’s program of farm collectivization, which has the state pushing things in entirely the opposite direction.
This reminds that I want to read Anne Applebaum's new book, "Red Famine: Stalin's War on Ukraine", which details the use of collective farming as a tool of mass murder.

Legibility, and the Choice of Model Parameters

After all of these interesting anecdotes rich with association, Mr. MacIver moves to his main point, which is that the attempts of tech companies to be disruptive, or to improve efficiency, can be boiled down to attempts to improve legibility. They aren't removing cruft: they're building new models of complex systems. They're choosing principal components – sets of parameters – that make it easier to understand the properties they want to optimize.

That model building produces lots of unintended – often bad – consequences.

... we’ve reduced the world to the small subset of it that we think of as the important bits, and discarded the old, illegible, reality as unimportant...
that legibility we impose often maps very badly to the actual complexity of the world. You only have to look at the endless stream of falsehoods programmers believe [in order] to get a sense of how much of the world’s complexity we’re ignoring.
Mr. MacIver next explains soft power, and in particular the soft power of disruptive technology companies. I took his words as an oblique reference to Amazon's business model, and free two-day shipping, but Mr. MacIver explicitly referenced other companies such as Twitter and Uber:
... soft power is power that you derive instead from appeal – People want to do what you want. There are a variety of routes to soft power, but there’s one that has been particularly effective for colonising forces, the early state, and software companies. It goes like this.
First you make them want what you have, then you make them need it.
The trick is to to basically ease people in – you give them a hook that makes your stuff appealing, and then once they’re used to it they can’t do without...
Pity they’re going to hike the prices way up when they’ve driven the competition into the ground and want to stop hemorrhaging money.
Sometimes it turns out that what we were disrupting was our life support system.

Admonitions

Mr. MacIver ends with some advice.

First he reminds the partisans of the old Mac / Windows wars that they both have it wrong, and that Richard Stallman's painful prescription is probably, by far, the best.

Then he advises that anyone who wants to develop a new, disruptive technology (and who is not cynically trying to control some large portion of society/industry, as I believe Mr. Bezos and others are) should begin with user research. "If you’re going to be forming a simplified model of the world, at least base it on what’s important to the people who are going to be using your software."

Finally, he practically begs us all to stop relying on ads. You should read his advice for yourself.

Thanks, Mr. MacGiver, for such a wonderful and thoughtful talk.

2017/10/17

The Founding Fathers designed impeachment for someone exactly like Donald Trump - The Washington Post

The Founding Fathers designed impeachment for someone exactly like Donald Trump - The Washington Post: in which Barbara Radnofsky lays out the many causes for which a properly functioning Congress would already have impeached Donald J. Trump.

Scores decry new science standards | Northern New Mexico Education | santafenewmexican.com

Scores decry new science standards | Northern New Mexico Education | santafenewmexican.com:

More than 200 people showed up at the Public Education Department on Monday to voice opposition to the New Mexico STEM-Ready Standards, due to go into effect for state schools in July 2018...
By midafternoon, not one person spoke in favor of adopting the standards.

Want to deepen our democracy? Get ready. - The Washington Post

Want to deepen our democracy? Get ready. - The Washington Post:

The time has come to make our democracy democratic again. And now, no one can say that we lack ideas for how to do it.

2017/09/25

Nuclear Experts Head to China to Test Experimental Reactors - Bloomberg

Nuclear Experts Head to China to Test Experimental Reactors - Bloomberg:

“China by a very large margin is the largest market in the world for new power plants of any type,” said Charles W. Forsberg, a professor at MIT. “If we do not get our act together, the low-carbon energy business will be owned by the Chinese.”
Yup.