Overall

That’s a lot of conference for $85. (I went to all 4 days). I wish I’d gotten the $150 ticket in order to be able to go to the Open Source Law training all day on Friday. The convention center was a good venue for this size of conference. There are many pleasant food places a short walk down the hill. The convention hotel was not great, but it was very near the convention center. Some of the SCALE14x talks were “amateur” vs “expert” talks, by the definition “An amateur talk tells you all the reasons that something is interesting; an expert talk tells you all that, and all the reasons why it is also a terrible idea with many attack vectors, but possibly still the best idea.”

My notes below are in reverse order because that was the most convenient way for me to write them, so the first talk that I saw is at the very bottom.

Ratio: there were some women there! I’d estimate about 8% or so.

Booths: The exhibit hall has a lot of good stuff in it. The Facebook booth was the fanciest. Nagios had the best tshirts. There were 3 booths which were covered in technical books for sale. :) :)

Terrible ideas in Git

  • java tool bfg
  • git-annex (by Joey)
  • git-lfs (by github, pushing it hard)
  • tool “fuck” lol
  • gorse
  • mr for multi repos management
  • VCSH

Project Hosting 3.0

  • Very empty room, 10 ppl. It would fit 100 or more. :/
  • FOSS Hosting
  • ISC does a lot of colocation hosting
  • CircleCI, Travis, Drone.io
  • Ganeti “Ganeti is a virtual machine cluster management tool developed by Google. The solution stack uses either Xen or KVM as the virtualization platform, LVM for disk management, and optionally DRBD for disk replication across physical hosts.”
  • love-hate relationship with GlusterFS
  • Ceph vs GlusterFS vs Swift (S3)
  • Ceph vs GlusterFS and another article
  • Neutral CDN mirroring
  • Current FTP mirroring infra is not flexible enough.

Preventing Unauthorized Email Spoofing with DMARC

  • describing different types of spam
  • DMARC “Got the best legs at the moment”
  • identifier misalignment.
  • SPF Sender Policy Framework
  • DKIM
  • ARF = encapsulate RFC?22 to –
  • mailman
  • options for dealing with mailing lists.
  • this is a good talk for people who run mailing lists. send to cusj?
  • arc-spec.org
  • openDKIM
  • adoption coverage 85 ish.
  • used the ashley madison data set. hacktivists. lol
  • detecting fraud examples on your network. army sending western union mail lol
  • https://www.socallinuxexpo.org/scale/14x/presentations/preventing-unauthorized-email-spoofing-dmarc
  • JMWSecure

DNSSEC

Sunday keynote

  • Watched online / skipped. I think that this was a good recap of the relevant points.

Docker, Kubernetes, and Mesos: Compared.

  • “Software is a liquid. Liquids take on the shape of their container.”
  • Barreleye - 160 CPUs on one board.
  • Carina “Carina makes container clusters more accessible to everyone, from novices to professional developers.” advertisement?
  • Rackspace Magnum
  • Why choose Kubernetes? “If you think that Google’s code doesn’t stink.” It is very proven in very production.

A Cloud-based Architecture for Processing 3D Mars Terrain by Parker Abercrombie

  • his blog
  • “NASA and Microsoft have teamed up to develop software called OnSight, a new technology that will enable scientists to work virtually on Mars using wearable technology called Microsoft HoloLens.” source
  • Project: OnSight- JPL + Microsoft.
  • Mount Sharp- where the rover is heading.
  • todo: dynamic rover path background?
  • google map mars
  • derive range from camera to each point on terrain, get point cloud, make mesh geometry, paint images onto mesh.
  • Navcam (rover), HIRISE (orbiter- least good), Mastcam (color, science camera)
  • In-house pieline. (audience questions)
  • 430 scenes, new every day, size of image varies. 1 scene- 5gb, 1k images, 2.5hr runtime on one computer.
  • bursty workload
  • tools: jenkins, ansible, loopback(nodejs), angular, Bootstrap, amazon SNS, amason SOQS, …
  • in-house build tool / build manager
  • automate jenkins with groovy scripts
  • GPU on cloud machines is troublesome- had to fiddle a lot, didnt work like desktops.
  • Spot instances! 10 cents per hour. Spot instances, not on-demand.
  • more audience inturruptions
  • future work: ansible to manage worker AMIs. Split pieline into smaller services. (why?), more autoscaling and management
  • they want to maintain some level of cloud-independence.
  • 2020 next mars rover lands, gotta be ready
  • Might use spinnaker

Continuous Delivery of Infrastructure with Jenkins

  • Jenkins Infrastructure Lead
  • SPI “Software in the public interest” holds the trademarks for Jenkins and Debian
  • OSUOSL Oregon state university open source lab
  • Jenkins - Infrastrucutre, Project Infrastructure- buid, developer tools, release infra, 100MB a week for years. Release infra, project tooling.
  • “testing infrastructure roundtable”
  • serverspec
  • “Every time I touch production, something goes wrong.” - the main reason why people are scared of continuous delivery.
  • “pull the trigger” on a deploy
  • testing infra: rspec-puppet, serverspec,
  • puppet is a DSL that compiled to a directed graph
  • “most people!” not my favorite phrase
  • TIL Blimpfile
  • “RedHat said Hey, maybe you should write tests for your code, because we would like to use it and we would love if you don’t break it every release.”
  • Docker- the image becomes another THING with a lifecycle.
  • “The boundary between your config management and your dockerfile becomes blurry, and you have to decide.”
  • iptables, faustian bargain with Docker.
  • Jenkins uses Bind9 DNS and also
  • garethr-docker “Module for installing and managing docker”
  • It used to be: archive and copy artifcats. Now, with the pipeline plugin, add it to “stash”
  • “becaue e run our name servers, and I’ really embarrassed when we screw those up… we have to test that too.”
  • manual gate to production is still tricky.
  • r10k
  • beaker-rspec
  • nagios table, jenins table
  • jenkins-infra on freenode
  • todo irc phone
  • typer@linux.com jenkins-infra workflow-examples
  • hiera eyaml to do keys.

Securing Microservices

  • slides
  • microservices: pick the tech that is best for your part of the stuff. (functional area- shopping cart, payments, etc.)
  • room is pretty full, ppl standing in back. Non-useful packing algorithm. I’m glad that I got here early.
  • “This young woman in the front.”
  • KeyCloak github
  • Auth0
  • StormPath
  • OpenID vs Oauth and also this
  • “And I’m sorry if you need to use SAML v2”
  • finger vein authentication
  • AUdience questions are just so what I don’t even.

  • Very loud presenter, lots of attendees, not super maazing content that I can see.

Lunch at Amara, with spicy hot chocolate.

Nomad - An introduction to Cluster Schedulers (*walk)

  • @dontrebootme
  • Nomad by Hashicorp github
  • Service discovery- consul, Zookeeper, etc.
  • cluster schedulers- started (for him) with a render farm. Knows where there is room, places jobs. Keep state information.
  • Tools that do this: Docker swarm, amazon ecs, kubernetes (googlegke hosted), mezos/marathon, Fleet.
  • Features like rolling upgrades.
  • Amazon ECS - only runs docker right now. Simple to get started.
  • Kubernetes: Borg successor, lots of PRs, contributors, from google, many components to it. Has a noton of pods- group services locally by host.
  • Apache Mesos + Mesophere Marathon (or Aurora or something else). Two-level scheduling (what is this?)
  • Hashicorp Nomad. Integrates well with Consul. Components: server, agent, job, task.
  • Questions from the audience
  • live demos (pre-scripted)

Keynote Saturday (10am)

  • I skipped most of it to go do errands, then watched the rest via livestream. Meh.

Bad Voltage

  • Some technical difficulties, then tech-ish comedy. I left early.

SCALEUp talks

  • “The FOSS Awakens” “Don’t be slagging other people’s projects” “Even if we could clone core contributors, we don’t want to.” “In our universe it’s illegal, but it’s also a bad idea.” “Evil Mistake Number Five… is being evil.” #scale14x
  • “Building Oracle in LXC in Ubuntu” ksplice. The war against slides.
  • Lucy Liman “GIF analysis” spelling it. LZW lossless compression. TIL “Blue” humor is a name for Off-color/vulgar/crude. GIFs are a way to express emotion… “In order to find this, I googled shark cat roomba duck, and google totally knew what I meant.” Types of humor- physical, crude, pun, (some more)
  • Don Marti “Saving Advertising” Aloodo There are important cultural works that were paid for by advertising, and we need to let that still happen. How did we get here from there? /
  • edunham, Where is the cloud QEDunham cable map “Put your data in the good old USA, where the unknowns about government spying are actually known.” #scale14x
  • Dave, Redis Labs, redis functions. Developer advocate… HyperLogLog hset “good C” redis
  • Julie Gunderson “embraced my sparkle” Words have power over how we see ourselves. Evolution of princesses over time. Overview of princesses.
  • Justin Cane “Choosing a framework” pedantry is alive and well in the world. “What is the official definition of a tool?” A tool cannot be consumed, but it can break. “When you screw up your whole project with a tool” So… like. I am not sure that emailing someone under
  • Cory Quinn “that time my boss destroyed a cubicle.” “You have all heard about Docker, because the first rule of Docker is- never shut the fuck up about docker.” “The first title of my talk was Docker is Horseshit, and the organizers asked me to change it, and then they asked if I wouldn’t mind changing it back. “ “And somewhere along here, we forgot to build roads.” Hasty Pudding lol.

Notes on lightning talks.

  • Mics are hard. The war to make your own slides work.

For the rest of the afternoon Friday, I wandered, rested, went to the hallway track. Went to get snacks for tomorrow, got coffee at a local place.

Config Management Sucks (*run)

  • “Thanks for coming to my group therapy session”
  • @rothgar
  • Disney animation! Tool hoarder
  • Best disclaimer!!! lolll. Second talk with a point against Docker
  • yay slides!
  • jinja2 like erb but for python
  • “Rub a little crypto on them and stick it in a file, and we’re good, right?”
  • Puppet is every fifth tweet on my timeline, so let’s use that!
  • How did I get that there? I need config management to get that place
  • “True story, the very first ansible cookbook I ever wrote was deploying puppet to my 7000 nodes” #scale14x
  • puppet syntax
  • eyaml encrypted yaml
  • mco plugins most popular: ping
  • ansible, salt
  • email to excel to state
  • waffles -config management in bash. other thing too.
  • why use these tools? Why better than waffles? “Meat space people don’t scale.”
  • scale, consistent (even if broken), reliability, community (“Use these tools because you will be an angry sysadmin if you don’t.”)
  • j.mp/cm-sucks-scale14

Fun times- before the config management talk, I talked to a dude from Procore. they have a rails monolith. Talked about plangrid, redis in use, gotta switch to resque for threading? Deciding to turn networking mode on is pretty cool.

Friday at the end of lunch, I met some cool people. Julie(?) and Austin, and she was working on her dissertation about Doctorow and she talked to him and got a point corrected :) :)

Open Source tools for distributed systems administration - Elizabeth K. Joseph

  • git
  • remote working!
  • 1 of 9 ppl who have root on all the things
  • Not stuff with bug reports
  • CI system - bash and python. Garrett code review, Zuul gatekeeper, Gearman distributes across Jenkins servers.
  • puppet! puppet linting
  • beaker-rspec “can-deploy tests”
  • checking irc permissions, checking xml syntax
  • asterisk server
  • so much “guys”
  • puppet vcs module
  • cacti openstack
  • puppetboard had to get rid of some things to make it safe in public
  • radical enabling other people debug all your stuff.
  • OpenStack CI
  • “Not logging into servers is really nice, because my ssh key lives at home, and that’s far away.”
  • “Complicated ugrades and migrations can’t be done via pull request”
  • “We document every command, because even if it’s easy, even if it’s obvious, when you’re in the moment, you forget how to run mysql commands.”
  • etherpad
  • secret git repos
  • “passowrds are not open source, but they are still in git.”
  • “We don’t really use phone or video, because… we don’t like it. We’re all on irc all day.” openstack-infra
  • every siz months, openstack summit. culture decays over time if you don’t see each other in person. helps you connect and work together. really curmudgeon on irc, but when you go out for drinks, everything is ok.
  • timezones… the only thing that weve found to help is getting more people in the timeone
  • “If anyone’s bored or wants some experience with ops, come hang out” jobs via being needed.

Joel

  • Hopelook (HauteLook?), PHP
  • “Burn the stump!”
  • Jmeter, Gatling, whee
  • caching proxy like varnish
  • bare metal php lol

  • tune-d for sensible defaults

Verizon dude

  • “We cover more of the stack than most companies”
  • Their recruiter

TDI Test Driven Infrastructure

  • describing agils stuff, devops, continuous delivery. Nervous, reading his notes, full house. Poor dude.

Make devops Great Again (10pm)

  • tool talks vs people talks
  • Velocity 2010 talk by Velocity 2010: Andrew Shafer, “Change Management: A Scientific Classification”
  • Andrew Clay Shafer
  • “This is old-school demagoguery, all the way back to the greeks”
  • Dunning-Kruger syndrome
  • flicker screen :/
  • “Everything looks like a shitshow from the inside”
  • “As long as we’re going to play capitalism, we might as well play to win.”
  • “If you have gained information about something, but you haven’t changed your behavior, you didn’t actually learn.”
  • “We have this intractable problem that we’re not able to solve, because we made bad decisions over here, and we’re not willing to revisit them.”
  • “How do we get to devops without actually changing anything?”
  • “He automated stuff and didn’t get much out of it”
  • “It is often an antipattern to automate what already exists. You have an opportunity!” “It is a smell if automating something is hard. Maybe, if you are fighting the technology, there is a better way.
  • Culture, Automation, Lean, Metrics, Sharing #CALMS / #CAMS #scale14x
  • Tools? Who cares, ok fine, I can make computers do stuff.
  • “This deep stack, that no one can understand fully”
  • What Devops Means To Me
  • “By playing the song, we should get better. Skill acquisition!”
  • “I was cringing when he said crapwork. That stuff is super necessary.”
  • Organizational learning: deliberately doing things to learn, questionairre, everyone encouraged to question, regardless of their level in the org. dept mtgs w/ no leadership other than engineers. Will they riot? ANgry lead engineers! they don’t
  • “There’s nothing in the CAP theorem that actually talks about computers. When you have a company-wide meeting, that’s a global write-lock on your organization.”
  • “Let’s just use Docker! Or whatever tool. Everyone wants to take the blue pill”

Doctorow’s Keynote (Friday morning)

  • he just moved to socal
  • tshirt: feacebook neteng as32934
  • science vs alchemy
  • rigor! “peer review- where your friends gently point out your mistakes, and your enemies tell you what an idiot you are to have made them.”
  • no seret math for building
  • “A building today is just a computer that we put our bodies into.” - @doctorow #scale14x
  • “Our cars are computers that we put our bodies into.”
  • World’s smartest thermometer- “we now have computers up our asses.” lollll
  • https://en.wikipedia.org/wiki/Pathetic_dot_theory
  • code that has DRM in it
  • DMCA laws
  • bill 92a, terrible- new zealand
  • mistakes in 1989 excusable as lack of foresight- in 2011, it’s criminal neglinance. / ignorance
  • printer that refused to let you refill your cartridge
  • insecure door openers
  • “When you put this thermometer up your ass, you have a copyrighted work up your ass.” #scale14 .@doctorow
  • “DRM has never been good at fighting piracy because of basic cryptographic reasons.”
  • “When you hide keys in your adversaries equipment, you lose.”
  • “No one has never bought something because of DRM.”
  • VLC and handbrake to record Netflix lol. No capital, no trade show booth for CES… htcp, downsample for older devices- now being sued. It’s a compeition supressor.
  • “That has an address that you can be sued at” … diagnostic tool :/
  • “But no one has ever argued that with few enough eyes, bugs are shallow.”
  • “DRM bans disclosure”
  • amazing analogies about ring 0 and -1
  • “Makes these devices into reservoirs of vulerbilities”
  • section 12-1 of the DMCA- copyright office held hearings- automotive security-
  • ifixit- 3rd party manuals ftw “the guy on the corner, the woman on teh corner” John Deere tractor, wouldn’t start, copyright office for an exemption, torque sensors, accurate soil surveys. that data is not copyrightable. can’t
  • “But thats just the moustache twirl. there’s a full on fingertips coming.” futures market with this data… zomg
  • type one diabetic “do things perfectly over and over agin especially when their blood sugar is too high or too low”
  • “festering vulnerabilitie in them” medical devices.
  • “tailored access operations unit NSA- skymall of nsa”
  • “NOBUS no one but us”
  • this speech deserves a transcript.
  • “And this is why I went back to the EFF”
  • subprime car loan - http://www.wsj.com/articles/total-u-s-household-debt-rises-to-12-1-trillion-in-third-quarter-1447948826
  • Changing the ncu of your phone lets you bill to someone else, duh
  • prothetics lab at MIT Hugh Herr lab1 lab2 lab3
  • “Treading water is a necessary but insufficient precondition to being rescued”
  • How to litigation harden your work
  • “Target-rich environment” cracking John Deere tractors. The scientific enlightenment.
  • Every week and month from now on, there will
  • Terrible things that you can do- nest
  • sensitizing people to notcing this in the world.
  • “There are very few of us who can claim to be pure.” Comcast, Apple, … “Every vegetarian eventually meets a vegan.” Instead of being pure, tithe. #scale14x .@doctorow
  • yay MonkeyBrains in SF!
  • How much money do you spend every week, every month, to
  • “Write that check every month” lol. EFF campus network, students, CS students.

Linus Sucks (Thurs eve 8pm)

  • GNU disclaimer
  • potato copier
  • mayonaisse! mutant potatoes
  • Wayland
  • Mir
  • “I love Fedora because fedora people are fun to poke. Like a hornets nest, but the hornets are rainbow colored”
  • systemd
  • “sucked” is an interesting word.
  • “Then the systemd creators did something dark. They started ADDING shit. How much shit they done added!”
  • “But it’s cool if the dad helps, right?”
  • “People care about iPhones a lot. This graph looks like a stegosaurus.”
  • “He put hishands down so fast.” Making it clear that this audience doesn’t like Bieber. Norm clarification.
  • Distrowatch
  • Stallman video clip
  • TIL Hannah Montana Linux

postgres security

  • people who don’t understand, barely listening.

Thursday, 2pm, debugging Postgres

  • @jim_mlodgenski
  • cto, openscg.com
  • {pgconf NY](http://www.pgconf.us/2016/)
  • he’s using fantasy football as an example- discussed how he had to explain his premise when presenting in Europe.
  • From his graph, it appears that PL/R triggers are slowest.
  • Code of what he’s using: git clone git://git.postgresql.org/git/pldebugger.git
  • live demo lol
  • Don’t put functions in where clause, unless theres an index. Will kill performance. Calculate on insertion rather than extraction. Spend once, read many times.
  • How many times a record has been read? (mi)
  • track_functions - populate system view pg_stat_user_functions
  • “alter system set” changes the config file so that it will persist, but requires a restart to take effect
  • plprofiler and pldebugger use the same hooks, and only one of them can run at a time.
  • I asked a question! How often you have to change it so it won’t break, how to test the new versions… looks like very academic software.
  • plprofiler

Notes:

  • Juju
  • Todo me- implement a tiny code debugger from scratch- in ruby, for ruby? p timestamp right before, after each line, with line number. _line num A for before, B for after, timestamp”
  • Chef room was very full
  • Always use long options in automation, never short ones. short ones are for interactive mode. Because it’s easier to read. (this could be a blog post) -n is –bindir in Gem (as in gem install), people rarely use long options for rsync but should!. Also “test” - in bash scripts has no long options “because fuck you, that’s why”
  • LPIC-2
  • Write The Docs
  • TIL Pulp “Pulp is a platform for managing repositories of content, such as software packages, and pushing that content out to large numbers of consumers. If you want to locally mirror all or part of a repository, host your own content in a new repository, manage content from multiple sources in one place, and push content you choose out to large numbers of clients in one simple operation, Pulp is for you!” by RedHat
  • mars.nasa.com
  • guy with zootopia tech animations shirt :)
  • I wish they would not photograph the audience. I am guilty too.

Facilities

  • Power in strung under every few rows of chairs in the talk rooms. :)
  • Wifi worked well!
  • Hotel gave me a not-great room which was also not what I asked for and called it an upgrade because it has a noisy balcony that other people can see into. Also then the pipes int he bathroom leaked a lot and made the floor wet. I got a partial refund.