OpenStack Summit

I  had the opportunity to attend the OpenStack summit this week. Below are my notes from the sessions I attended.




The Future of Open Stack Networking

  • NAAS (Networking as a Service)
  • Network abstraction to tenants, a set of api’s and a couple of plugins.
  • Plugins to be more like drivers.
  • Started off just being L2 connectivity. Wants to have higher level functions and policy. Multiple vendors working on it.
  • Intent is to take a very high level to networking. SDN (OpenDaylight)
  • What are abstractions in Neutron?
  • LBAAS (Load Balancing As A Service) Deploying LBASS to scale properly, companies have pains doing this.
  • Make sure you have use case documents prior to deployment.
  • What does Adobe’s use case document look like?
  • ML2 plugin with openVswitch and Linux bridge agent.
  • What is the thinking behind virtual networking policy
  • Group based policy, building a new set of api’s
  • The whole purpose here it so enable quick deployments, ability for developers to create connectivity between applications they are developing.
  • This storage node needs to communicate with this other node (VLANs, subnets, ect.. will automatically be created)
  • Policy time of day (from 6 – 7 I want X policy deployed)
  • Security policy boundary, security policies based on application and pushed down to underline architecture.
  • How do you bootstrap the entire system, including networking? Ability to bootstrap system from scratch.
  • Look into VXLAN gateway.


An Overview Of Open Source Back Ends for Neutron

  • Neutron ML2 Plugins (OVS or Linux Bridge)

o   Drivers go through Testing!!!

o   L2/L3

o   IPAM

o   East/West Routing

o   Floating IP’s

o   Layer 3 and DHCP agent

o   Load balancing, VPN, firewall. Service Plugins.

  • Neutron Server

o   ML2 and L3 Base Plugin

  • Installing Via Devstack

o   local.conf

  • enable service neutron ql3 q-agt g-svc, q-dhcp q-meta – ./
  • Neutron’s built in solution: limits

o   Salability limits of the L2 agent

o   Focus is not exclusively networking.


o   An overlay mechanism driver for ML2

o   Installs Proxy-Arp

  • OpenContrail

o   An extensible networking system covering two primary use cases:

  • Cloud Networking
  • Network Function Virtualization

o   Integrates with OpenStack Neutron.

  • Nova creates VM on compute host
  • Nova agent gets acquired by plugin
  • Contrail configures networking

o   google: using devstack plus opencontrail

o   Will go upstream in Juno

  • OpenDaylight

o   Working on a helium release set for the fall

  • Ryu Network OS

o   SDN Framework


o   Stand Alone plugin

o   Ml2 mechanism driver

o   Multitenant network (mac address, vlan, gre)

o   Ryu supports port binding (Bind Virtual Ports)

  • For Juno deployment, distributed virtual routing.


Inside the Architecture of Neutron


  • Neutron server

o   Has a relational database

o   Talks to Message Queue, used to talk to agents

o   Talks to L2 agents

o   Talks to DHCP agent

o   Talks to L3 agent

o   When running in real world you will have multiple agents.

  • Neutron server is a plugin.
  • Ml2 plugin, neutron db plugin v2.
  • Supports plugin extensions.
  • Monolithic Plugin
  • ML2 (Modular Layer 2 Plugin)
  • OVS Agent (Open Virtual Switch)

o   Ensures that the ports within the switch are logically grouped together. Tunnels traffic.

  • Linux Namespaces

o   Can talk to other services on the same node.

  • VPN as a service
  • Firewall as a Service


Troubleshooting Neutron Virtual Networks


  • Troubleshooting Process

o   Define the Problem

o   Examine the Situation

o   Consider the Cause

o   Consider the Solutions

o   Act and Test

o   Review Troubleshooting


  • Problem Situation (VM Does Not Get An IP)

o   ip

  • address, route, netns, neighbor etc…

o   ifconfig, route and netstat are deprecated

o   iptables

o   useful options -n -v (packet statistics for that rule, watch command) –line-numbers

o   ping, host, traceroute, tcpbump, ip neighbor, arp, arping

o   wireshark

  • OpenVswitch

o   ovs-vsctl

  • show
  • add-br

o   ovs-ofctl

  • dump
  • dump-ports
  • show

o   ovs-appctl

  • bridge/dump-flows
  • fdb/show

o   Port mirroring on OpenVSwitch

  • create virtual pair with ip link
  • add it into vswitch vridge
  • create the mirror and mirror the packets from eth1, br-int, patch-tun.
  • delete mirror when finished

o   Flow Tables

  • br-tun flow table
  • show what table by looking at resubmit

o   Read OFCTL man page

  • Neutron Debug Command

o   probe-clear

o   probe-create

o   probe-delete

o   probe-exec

o   probe-list

o   ping-all !!

  • Neutron Troubleshooting

o   Gather

  • mac and ip add of vm’s, dhcp server, router
  • mac and ip add of data network nodes
  • set the neutron service to log at debug level

o   where is the problem?

  • one tenant or all
  • one network or all
  • what protocols are used
  • L2 or L3

o   examine/locate

  • look carefully at what is happening
    • typically insufficient time is spent here
  • isolate to tenant, network, VM, compute or network node

o   consider causes

o   need more data?

  • repeat steps

o   test

  • only adjust one thing at a time
    • if that did not fix go back
    • keep a log of what you have tried
  • repeat
  • Network Monitoring

o   traffic levels

  • add sflow to openvswitch
  • watch for failures


  • Notes:

o   udhcp -T 1

o   tcpdump -e

o   watch “ovs-ofctl dump-flows br-tun”

o   ip netns (for namespace)

  • ip netns exec (namespace) tcpdump -l

o   ps aux | grep dnsm

o   restart neutron-dhcp-agent





BigSwitch Networks


  • Talked about bare metal switches

o   A fraction of the cost of regular switches

o   Could start off with 2 spine and 2 top of rack for test

o   Same throughput and subscription model as high end Cisco switches

o   Can utilize switches in a monitor mode (span aggregator) at a cheap cost

o   If we are looking at building a cheap deployment of openstack I would suggest we try out some of these bare metal switches

o   Controlled by a SDN controller

o   Total separation of Data and Control plane

o   Full redundancy, network can still run while any one piece is offline.

o   Contact Michael Centrella


Juniper Networks


  • Talked with Juniper about their current SDN and OpenStack solutions


o   Provides switches and routing instances for openstack

o   Juniper router replaces Linux namespaces

o   When you configure new networks in OpenStack dashboard changes are pushed through the network to Juniper Physical Routers and Switches

o   Juniper was built on BSD so it was easy to port their code into openstack

o   Will natively support Jumbo frames

o   I would suggest meeting with them about their OpenStack Solution and the neutron plugin.


Deploying the Neutron Server


  • Packstack to deploy via command line.
  • Ml2 Plugin

o   Supports an extensible set of network types, each implemented as a TypeDriver

o   Works with a variety of virtual networking mechanisms (simultaneously), each supported via a MechanismDriver

o   Supports multi-segment L2 networks

o   Supports heterogeneous network configurations


Working with Neutron Security Groups


  • What are security groups

o   Security group rules allow us to define access into and out of instances.

o   Default is no traffic in and all traffic out

  • Can control via port, ip range, or another security group
  • Security groups are implemented using security group rules on the hypervisor (iptable rules)
  • These rules are visible in the host operating system.
  • Nova commands to create security groups work.

o   Neutron security-group-rule-create

o   Nova secgroup-list

  • Security groups should have unique names

o   If you don’t use unique names on sec groups you have to use uID’s.

  • When you create a security group you can specify a CIDR range or another Security Group. This can allow you to create tiered deployments.
  • When you boot an instance up you have the ability to specify a list of sec groups you want that instance to be a part of.
  • Once the instance is up you can modify which sec group is attached to.
  • Neutron will not let you delete a sec group that has any instances attached to it.
  • Logging

o   Need to research iptables logging on instances


Meeting with Cisco, Ubuntu, and even Mark Shuttleworth!

  • Apparently it is a big deal to Ubuntu and Cisco that we are successful for this deployment. Mark Shuttleworth Founder and CEO of Ubuntu came to the meeting and dedicated Ubuntu to supporting us in this project.
  • Cisco indicated specifically that they will be ready to support 14.04 icehouse with their N1KV code.
  • They asked Arghya if he wanted to be a keynote speaker for the next OpenStack summit in Paris. We mentioned that Rick Brown loves to be the center of attention and would be a perfect keynote speaker.
  • Talked about bring some of their “Orange Boxes – openstack in a box” boxes to adobe for a 2 day training for Adobe employees on Openstack.




From Infrastructure Administrator to Cloud Architect – Panel of Cloud Architects giving advice


  • Brush up on your Linux skills
  • Brush up on how relational databases work in Linux
  • Use Slideshare, Youtube, Blogs to learn what others are doing. You learn most form the community.
  • Think of things in a high level, there is a lot more to the architecture of OpenStack than just managing a vendor cloud environment.
  • You have to understand that vmware is fundamentally different, you cannot compare both.
  • Use DevStack learning tools to get your hands dirty.
  • You do not have to be a Python developer, but you should have an understanding of what it is and how it works.
  • OpenStack is NOT a free version of vmware, you cannot use the same processes.
  • Need to learn skills on how to talk to developers.
  • Not do you just need to know how to operate the cloud, but how to operate it efficiently.
  • Its not just Linux, network, storage, you have to understand the app’s as well.
  • Access work loads that you have in your environment today and in the future and figure out which architectures will work for you.
  • You always want to be outside your comfort zone.
  • Need to be able to see the whole picture. There are over 500 configurations running in OpenStack today. You need to be able to see what configurations will work for the company you are working for.


Meeting With Cisco (Balaji)


  • Going to meet up the week after Cisco live
  • Arghya and I need to provide documentation to Cisco
  • I will provide a document to cisco that provides network documentation and network specific requirements for this next build out.
  • Look into Lehi and verify that version of code is sufficient to support OpenStack
  • Follow-up on Cisco CSR virtual router


Deploying OpenStack with Cisco Networking, Compute, and Storage


  • ACI (Application Centric Infrastructure)

o   9k’s can run in ACI mode

o   VXLAN routing

o   Define policy in an abstract sense

o   Policy can be distributed across the network

o   APIC pushes policy

o   Designed to scale to over 1million end points

o   What is application policy

  • Helps an application developer to tell network what to do
  • End point groups (kind of like a vlan, a way of tying machines together with similar requirements)
  • Wrap endpoint groups in contracts (rules and actions)

o   Opflex

  • Distribute policy using aci framework
  • Can live ontop of openvswitch

o   Tying together physical and virtual infrastructure (service chaining)

  • Simplifies operations
  • All hardware endpoint encapsulation done in hardware at tor switch

o   Application acceleration

  • UCS (Cisco Servers)

o   Both blade and rack mount

o   Unified Management (UCS Manager)

  • Sits of fabric of ucs itself
  • Service profiles
  • Bios capabilities
  • Scalable compute and storate

o   Python based SDK

o   Focused on Automation to scale deployments of openstack

  • Cisco Nexus plugin for Neutron

o   Supports all Nexus products

o   Nexus 9K promoted for high performance

  • Native programmability including containers
  • All full line rate (now back plane)

o   Automated VLAN provisioning

o   Linux Layer 3 agent can be replaced by Nexus TOR switch.

o   Nexus 1000

  • Can do service chaining
  • Nice support for VxLAN

o   CSR (cloud services router 1000v)

  • Replacement for linux namespace
    • Eigrp, netflow, etc…
  • VPN as a service
  • Opendaylight plugin

o   UCS manager ML2 plugin

o   Innovation in ipv6

o   Group based policy abstractions






OpenStack Networking Hands-on Lab


Was a hands on lab to get experience with Neutron.




Get a lab:

Access code: OSSATL14


# Source openstack credential file

source demo-openrc


# Create private network

neutron net-create private


# Associate subnet

neutron subnet-create –name private_subnet private


# create router

neutron router-create myrouter


# uplink router to the public internet

neutron router-gateway-set myrouter public


# uplink subnet to router

neutron router-interface-add myrouter private_subnet


# create security profile for jump host

neutron security-group-create jumphost


# Add rule to allow icmp in

neutron security-group-rule-create –protocol icmp jumphost


# Add rule to allow ssh in

neutron security-group-rule-create –protocol tcp –port-range-min 22 –port-range-max 22 jumphost


# Launch jump host:

nova boot –image cirros-0.3.2-x86_64 –flavor 1 jumphost –security_groups jumphost


# Determine ip that jump host was assigned

nova list


# Determine port-id

neutron port-list — –fixed_ips ip_address=<ip_address>


# create floatingip

neutron floatingip-create –port-id=<port_id> public


# test ping/ssh password is cubswin:)

ssh cirros@<floating-ip>


# create web security group

neutron security-group-create web


# allow tcp 80 in

neutron security-group-rule-create –protocol TCP –port-range-min 80 –port-range-max 80 web


# allow ssh from members of jumphost

neutron security-group-rule-create –direction ingress –protocol TCP –port-range-min 22 –port-range-max 22 –remote-group-id jumphost web



# boot two webservers

nova boot –image cirros-0.3.2-x86_64 –flavor 1 webserver1 –security_groups web

nova boot –image cirros-0.3.2-x86_64 –flavor 1 webserver2 –security_groups web



# test that you can ssh to them from jumphost and spin up a dummy webserver

$ ssh webserver1

$ while true; do echo -e ‘HTTP/1.0 200 OK\r\n\r\nweb_server1′ | sudo nc -l -p 80 ; done &

$ exit

<back on jump host>

$ ssh webserver2

$ while true; do echo -e ‘HTTP/1.0 200 OK\r\n\r\nweb_server2′ | sudo nc -l -p 80 ; done &

$ exit


# curl webserver1


$ curl webserver2




# create loadbalanacer pool

neutron lb-pool-create –lb-method ROUND_ROBIN –name mypool –protocol HTTP –subnet-id private_subnet


# Add webservers as memebers

neutron lb-member-create –address <webserver_1_ip> –protocol-port 80 mypool

neutron lb-member-create –address <webserver_2_ip> –protocol-port 80 mypool


# create health monitor

neutron lb-healthmonitor-create –delay 3 –type HTTP –max-retries 3 –timeout 3


# associate with pool

neutron lb-healthmonitor-associate <heath-monitor-id> mypool


# create vip for loadbalaner

neutron lb-vip-create –name myvip –protocol-port 80 –protocol HTTP –subnet-id private_subnet mypool


# associate floatingip to vip

neutron floatingip-create –port-id=<vip-port-id> public


# requests are now loadblanced over vip ip:

curl vip-floatingip


## Firewall as a service


# Create a firewall policy

neutron firewall-policy-create default_policy


# create firewall and attach policy

neutron firewall-create default_policy


Blocks all traffic


# create rule to allow http 80 in

neutron firewall-rule-create –protocol tcp –destination-port 80 –action allow –name allow_http


# add rule to policy

neutron firewall-policy-insert-rule default_policy allow_http


# curl works again


# test loadbalacner timeout

nova delete webserver1

curl vip-floatingip only returns webserver2