Pisces Quick Start Guide

These instructions have been tested on fedora running kernel version 3.14


  1. Before you start
  2. Cloning the repositories
  3. Set the configurations
  4. Building
  5. Pisces Enclave Control
  6. Palacios VM Control
  7. Frequently Asked Questions
  8. VM Configurations

Before you get started

Is your kernel configured to support memory hot removal? (CONFIG_MEMORY_HOTREMOVE=y)
Does your kernel support hot plug CPUs?
Do you have your kernel source located at /usr/src/'uname -r' ?

Do you have syslinux, syslinux-devel, glibc-static, glibc, and ncurses-static installed?

Cloning the repositories


git clone http://essex.cs.pitt.edu/git/palacios.git


git clone http://essex.cs.pitt.edu/git/pisces.git


git clone http://essex.cs.pitt.edu/git/petlib.git


git clone http://essex.cs.pitt.edu/git/xpmem.git


git clone https://www.github.com/ktpedre/kitten

Set the configurations

*Default settings work unless otherwise stated*


make config

System Architecture:
>3. Multi-instance environment (CONFIG_PISCES=y)

Include Palacios virtual machine monitor Y
Internal/External Palacios library
> 2. Link in Palacios as a self contained module (PALACIOS_EXTERNAL=y)

Path to pre-built Palacios tree (PALACIOS_PATH) /path/to/palacios


make config



(1) Build PetLib
make all
(2) Build XEMEM
cd mod/
make all
(3) Build Kitten
make all
(4) Build Palacios
make all
(5) Build Kitten (again, to link Palacios)
make all
(6) Build Pisces
make XPMEM=y

Pisces Enclave Control

Launching Pisces Enclave

change to xpmem directory
sudo insmod xpmem.ko
change to pisces directory
sudo insmod pisces.ko
cd linux_usr
sudo ./pisces_load /path/to/vmlwk.bin /path/to/pisces_init_task 'console=pisces'
sudo ./pisces_launch [-n <numa_node>] [-c <cpu_id>] /dev/pisces-enclave0

Adding Memory and CPUs to enclave

sudo ./pisces_add [-n <numa_node>] -m <num memory blocks> /dev/pisces-enclave0
sudo ./pisces_add [-n <numa_node>] -c <num CPUs> /dev/pisces-enclave0

Add PCI devices to enclave

sudo ./v3_pci -a <device_name> <bus>:<dev>.<fn> /dev/pisces-enclave0

Connecting to Pisces console

sudo ./pisces_cons /dev/pisces-enclave0

Palacios VM Control

Create the VM

sudo ./v3_create -b /path/to/vm/config.xml <vm name> /dev/pisces-enclave0 [config options]

The above mentioned config.xml is a .xml file that specifies settings for the VM. This includes the memory page tables, the CPU cores, the APIC, the PCI devices, the Timer, and many more. Example XML files are included below.

Launch the VM

sudo ./v3_launch /dev/pisces-enclave0 <vm id>

Connecting to the VM Console

sudo ./v3_cons_sc /dev/pisces-enclave0 <in>


Host locks up upon calling pisces_add

This is a problem which can occur without certain arguments given to the kernel. This may be attributed to an interrupt problem. A fix for this is to set the follow kernel arguments nox2apic intel_iommu=on intremap=off iommu=pt acpi_irq_nobalance in grub's grub.cfg file.

The pisces console says Enclave Created: 13 but the instructions are for enclave0

This error can occur for a number of reasons, including not have su privilges or using a bad pisces_init_task

I get the error "cannot launch VM" after running ./v3_create

This error occurs for many situations including a bad config.xml file or a bad pisces_init_task. More details can usually be found by running sudo ./pisces_cons /dev/pisces-enclave0

Error opening VM console while running ./v3_cons_sc

This can error can occur from not having the console correctly setup in the config files. See the config section for more information.

.xml config files

The config.xml is an important part of setting up the VM. The config file defines the devices available to the VM, defines the VM's memory page tables, and configs the CPUs.

*all of these .xml files need to be customized to fit your setup and due to active development this section may not be up to date*