Controller Design--Full State Feedback Controller
From System Modeling section, we got the state space model. We are going to use Matlab to design a full state feedback controller. Here is the Matlab code for this:
To help determin the feedback gains, we use a method call LQR--linear quadratic regulator. Refer to line 26-28, it will result feedback gains K and the system eigenvalues. When we tuing our feedback gains, we play with Q values and R values. In Q matrix, each diagonal element relates to one state, the first element related to the first state, etc. The larger value assigned, the greater effect on minimizing the variation of that associated state. R is associated with system input, the larger value of R, the greater effect on minimizing input value.
In line 33-35, a simulation is run and will give the below result:

This plot will help us determin whether the system performance looks good enough to give it a try on the real system. If you find Out(2), which is the plot of pendulum angle has a fast response and goes to zero in less than 1 second, then the feedback gains will probably works fine. Several trial and errors are unavoiable before you finally find the gains that working well.
Controller Design--PID Controller
PID controller is a very common-use controller, here is a video of applying a PID controller of this rotary inverted pendulum:
So, give yourself a try. The final programs are available in DOWNLOAD section.