Commit 88140a20 authored by Mathieu Reymond's avatar Mathieu Reymond
Browse files

minimal code cleaning, basic documentation

parent b28a1fd1
[submodule "Libs/darkflow"]
path = Libs/darkflow
url = https://github.com/thtrieu/darkflow.git
<mxfile userAgent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0" version="6.5.8" editor="www.draw.io" type="github"><diagram name="Page-1">7V3dc6s2Fv9r/JgMQnw+JrlJ25nbNtPc2e4+3ZGxbNNi5AU5cfavXwkQRkI4si0c56Z+iRFYwPn8nXN0lAm8W21/KtB6+SuZ4WziOrPtBH6ZuK4LQcz+8JHXeiQCoB5YFOmsHuoMPKX/w82g04xu0hkupQspIRlN1/JgQvIcJ1QaQ0VBXuTL5iST77pGC9wbeEpQ1h/9M53RZfMWbrAb/xmni6W4MwiaF56i5O9FQTZ5c7+JC+fVpz69QmKu5kXLJZqRl84QvJ/Au4IQWn9bbe9wxmkryFb/7mHgbPvcBc6p0Q8c2DwIfRUvj2eMFs1hTnL257Z6I8x/47CjJV1l7CtgX/E2pf9uhvn3//Dv15Af5bR4rU5dh8AXA/V5B4Ri4BEX6QpTXDST/IUpfW0EAm0oYUOkoEuyIDnKvhKybm7cf9Hm3UuyKZJWnBoyU1QscHNdFNZj/DU7v2zo8xMm7HGKV3ZBgTNE02dZJFAjWYv2uh112ZeGwHpiu/UUzyjbYCEeQUY5RdnswYJW71WPzAl7ry5bgv9uiDhxVVYEumEXuN56uzspZnlcohKzs6C+6PfpX1xLmD4ySic0Jbm4zbRQb8xeor53b7j7kIrIULylsmCUtCB/4zuSkWInRvM0y5QhlKWLnB0mjI9cCG6fcUFTpok3zYlVOpvx29y+LFOKn9aoYu4Lszv7pIDPgrd72ducvfKEkDS2Corjl53mQ6Gvy67WiwtPkQloIBOCm27NzTuSJ3jN2ZlhVORpvtjLzYPYVmAmWmhaXcDVcU3SnFav599O/C+cYUwpa/GrfiD4l+E51XCPcoW9LRnX2GN+q7T3yrPDOBApfHN9M775FtjmHcA2WLPtsSCzzdvK96OyKzJgV8sa2+zyNexSiJullT2q7ZZw+vzdZ2nR2Ez4hTkXPt5lwbrjwLj3nzK71nFqGpawOfjN7lrs4lizZBKF3bhvyICGwGFkgcLBZ6BwCN6RwiA8CamZAqsW0V3H4rAGba4fNMcqZutAPRBGMtSLgDcE9YwgXNgHcMB7LwAHokGzb4rXgBav3WWoLNN5yp964rJh50uF1AZBmQ3PYIy8MjTF2W0b2QgEt4tttLwMB7Srjdqap5t0Ix+d1jGpgj6UFO+qsS7GHG4mf+RU6FxC5vOSiZUqAu0zGElFOCgU02NR/YCU/Pbb2yh9OigkbazJ5WGGyqXOTGhxeyUAjwxHNEa6JzJflQumhFKyGgYYS7TmD7XaLnj+4HpVJghfJxnZzK5LXDynCf4+R9MiTZSgYcLR+e39g2vJY7oKKAn9uG/Q9djfRjwYn2TRO2Y3dtqAuonAnYNsvs487xyBU8fvrSfwfDjgCYxMuoh5xrXpisYLFCrz23MVx1w/Z/Ojk6wCdEbyFY1n4FeoYcQNpUU63VBctn5kLZTy43oSoSane5IrhmIcGMiuxD/NlYip5VmFPFt1NEAXdfaNvGRaF8n6eoWSJYPD30W64Po5LZlIfEfrVGa1Ymo9GIUPnMFsqlnK+NE9F0Tw/mbIYRzpF3qCZMPGAwW0+64GtOsCTwBtoPZYw7NR0MG/Kqay0zePv5wCE/oWwAYXQoUJsM+ESMMDGyxofdC5Aie36y7Nk9WiRtL1i9Zz1QPWy1VyB4Ga27TnGIXRsu4YvxUo/dB+bkirjvFzEISyR7qkiAk4feRrAnMdv00mNDA3jofg6+FqG0oot2WRonvH6XHtusfXWz/yx9JbAPqK+w2v1uwF8KWql7EqKVyt5XOffgEY+hLlr7zTFGwrTyNmBcoMVvTPHSs26UGbX1a80u46lW2uKlXODFGkByJmllsgXD5PSUnBBUJXG7QMYNQyoaepX7QVQeswUgQtp5bpr6MgkmL5yJWL9bJ9rZDMIZb0qISAiGu6BlMA8rGRj6uwNQhdeYr6oaxkBIYThadp3U2eE1pZ4Q+LfYR828gWO54CfcAFQR9twX8QCoGz2DYQKKYN9IMzqIvOQGwjPBuupVuOkJ9ytC6XhM8wL8iKiwSZVocJWuEC2Y2aP1Bt3lUEAGrqmtrCppo+PYr/BsV54fOT1yxlqlG8rRZKKot/Hh7sUMtTchme1y8aeBpiBTZKBjofohCrQwUdkVC5rovi83TLjUxbjUlLcreZ4j7xHh44+SyJWhzKxIs0iaCwTzzxs5OIpyvfqqgqy9J1iTtk2ayy+5ym9PVgoXtgnyCwZKPjSKYb9PoqqjPR/ul08wwWdxxAmQasWiGKAt4CjeMaKavo62KoC3PkgXdOR+6Dtyly0YY8CM9nyP1+MuVjG/JQAxrHMuS+AYj+KIY8CM5oyA2g1jsZcqgQxcyQuxaI4veJ8rRC1SWv7ErywumzXuLCUpbzYkC/6ys0d3s0B1qi2/AVfUjxlSdaeEzGbBmLyLgB+7HoHcv0jpywT29dAtEKvQ3ihnfRfFeBcDo3ok+r2ui+8A0ignchixfIdYAg7JNFeMEuVWwsNBZr4DpE+RXP0g1PmEz5ENNOAd9+KA31HFlDQ40XGs0iisLdyRn+WMrvi5rm8JptOeMfHpLxH6R5N7svWiPPkN2/KQr02rmgkTWVC4dlbCNd9GslHYmzuiuPNO15xi15NpRu3Hy+kGYraxmcdlmuqIo2lu/E2s8YpdVYF9dZkRamgaYCcYzhkJs9Qr9rIzgPXNlOwHYlxqgdvDrjEUW2jYdxk5NuMd9AhgMllByY3rAAH/Z4Lm3eR5SjTqFKqMuEjVLA+X2NC8TJ+ukKNSJmiKARX20AkvCAdN7e1cZVX8d3dXhvwuWHWHkcQDnTo1147HkarbSRnxbrGM7QltT2o1tfefwRVFNwO5CDBx8arg+yEjwYhPdt6r1ODb9LWAsjIBEpEkamSySRhV/KO6xYoJJBtH8JVGq7EVoqaQyH1p1bodLZOha+ciV1nVuy/bRWw1WaU+K4nxQMRzIa8dlw2x/Vg7vO9BNz2oPhm5weC8rFBlDuUutlLpAX88Wa0GasellsUGe8qIq2upArBv3i4lgV7digO/GiK9rtkhrhdjWr4MaqaMcGmQWcz2743nZc2Kr9IpLJUKOM57SruqseNNHCvb/PO/A6OaCrbt/3ccu6dbuvCcD3ZvKmW+jYsyvQiSnAwJUVJjRsCO9N5KuaJ7qW7a8jb/ce+KeDThHCVossrCKHsVKCu6RV5MDRmAfr2zbKuzZOzrI547vldoHT919/4EVackHlT84EtdO/dPnK0MqInX7SWNYGO91uojte2dbHsq6cLc7ZZcLoYL/bMdv2HNrvNk5awlc2T4k0G2WO1wEHnNP2Pxve1ywMB1DOQdvndFv/Ow10Q6jLyEQKsCR1EotFCqNvlaNAI1+FRjYRTT8z98TgrmR3E7JaIT72EUyvtc3PuOn1IqUp2FL5WzW+Si3QkvU9oL3nfWqmAxuUtq30b5o3z4p5G15mYjsbVze9fdI8XKw0/IaaOk0wUh6u3YropDC/dWT6bWYO3a/C1cDwZg+L94rKe832oKdjpnH5lZoS89VWF4tuDPQD8/uULrmHuTHQonPuma7xTD05GXZK6j56vqaLT5dntKNEJt0fprmy9qCz4UFn50NfxorB/o0PD1U9Hb7zB/DAeTTPVzvqDPeB6usdVIqYwBlT8XSxOy1S/MxXK/dXVooVl84cI7opcGnHx13OBja1juxDlQ50L2aDKHa4+5cn9eW7/ysD7/8P</diagram></mxfile>
\ No newline at end of file
Subproject commit 1dcd74e2ffea8f721607a57cdafbd4c026f57f36
# GarbageBot
# Setup
## Installation
......@@ -114,19 +114,52 @@ You can move the robot's arms, and click on execute to see the arms move to the
Programmatically, take a look at the [python interface tutorial](http://docs.ros.org/kinetic/api/moveit_tutorials/html/doc/pr2_tutorials/planning/scripts/doc/move_group_python_interface_tutorial.html).
You also need to run the `joint_trajectory_action_server`. However, with that alone, you get a time_out when running the `moveit_robots_test.py` code. One way to fix this (I still don't know if its the right way) is to run, in a separate terminal:
```bash
rosrun moveit_ros_move_group move_group
```
If you get an error looking like
>Robot semantic description not found. Did you forget to define or remap '/robot_description_semantic'? baxter moveit
try opening MoveIt with Rviz first, then close it and finally run your code. The semantic file is supposed to be created dynamically, but apparently this is only the case for the graphical interface.
If you get a boost error like this:
```
'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::lock_error >'
what(): boost: mutex lock failed in pthread_mutex_lock: Invalid argument
```
Ignore it for now, your whole script runs anyway and I could not find a fix yet (and btw, _just ignore it_ seems to be the main proposed solution).
# Garbage bot
Garbage bot sorts items, either left or right, according to their features. The features we used are *color* (red, green, blue) and *shape* (cube, sphere, triangle).
## Dependencies
The whole project is programmed in python (2.7, but _should_ work with 3+), making use of both *MoveIt*'s python interface, as well as *ROS*' python interface.
Next to that, you need to install a bunch of dependencies to run the decision trees and neural nets, as well as image processing:
- numpy
- six
- scipy
- tqdm
- sklearn
- scipy
- opencv-python
- imgaug
- tensorflow
- tflearn
- pandas
## Structure
The project is composed of 3 main parts:
- `decision_api`: used to train a decision tree based on a set of samples possessing each a specific *color* and *shape*
- `detecting`: used to train a neural network on images to extract their features. Provided is the training dataset used, as well as our model's weights. In `model.py`, you can either train the network, or use an existing one to extract features out of an image. The other files are examples that can be used separately: `take_images.py` will, given a set of features, take (for each arm) a bunch of images (that we used later on as training set). `predict_images.py` uses the robot to take a snapshot and extract its features.
- `planning`: this module focuses on the robot itself: moving the arms and taking snapshots. All this is done in `baxter.py`. If the robot runs in a simulator, a _Scene_ needs to be specified, which is done in `scene.py`.
All parts are merged together in `baxter_agent.py`.
## Executables
If you want to run a file using `rosrun`, two requirements are necessary: the file must be executable (`chmod +x <file>`), and the command must be specified at the beginning of the file (`#!/usr/bin/python`). We provided a bunch of examples to use, that can be executed using `rosrun garbage_bot <file>`:
- `predict_images` moves the robot arm to take a snapshot and extract its features
- `take_images` for each arm, snapshots are taken from random (but closeby) vantage points. They are then saved to the disk
- `example` moves an arm in the simulator to take an item and sort it
- `initialize` simply take a snapshot and save it to the disk
- `demo` runs the whole process. First, the robot will train, by having the human specify, for each given item, which arm needs to sort it. Then, the training phase ends, and the robot will sort the next items itself.
......@@ -36,6 +36,10 @@ class BaxterAgent(object):
return dm.predict_features(self.ir, img)
def _train_sort(self, arm_name, msg):
""" callback used to sort item. The specified arm will move to take a
snapshot, extract features out of the image, and add those to the learner.
Then move the item out of the way
"""
arm = getattr(self.baxter, arm_name)
if msg:
arm.move(arm.top)
......@@ -52,6 +56,9 @@ class BaxterAgent(object):
arm.move(arm.neutral)
def _train_end(self, msg):
""" callback used to end training. The learner will then use the samples
previously taken to train the model
"""
if msg:
self.is_train = False
try:
......@@ -63,10 +70,13 @@ class BaxterAgent(object):
def train(self):
self.drawTrainingUI()
# callback for left-arm wheel-button
train_left = lambda m: self._train_sort('left', m)
train_right = lambda m: self._train_sort('right', m)
# add callback on button click
self.navs['left'].button0_changed.connect(train_left)
self.navs['right'].button0_changed.connect(train_right)
# add callback on back button click
self.navs['left'].button1_changed.connect(self._train_end)
self.navs['right'].button1_changed.connect(self._train_end)
# blocking call until end button is clicked
......@@ -75,6 +85,10 @@ class BaxterAgent(object):
print('done training')
def _sort(self, msg):
""" callback used to sort an item. The left arm will take a snapshot,
extract features and send them to the learner to choose a side to sort.
The corresponding arm is then used to sort the item
"""
if msg:
self.baxter.left.move(self.baxter.left.top)
img = self.baxter.left.take_snapshot()
......@@ -120,9 +134,8 @@ class BaxterAgent(object):
rospy.sleep(3)
def drawTrainingUI(self):
print 'Garbagebot now training. Please choose a side to sort after putting down an object'
print('Garbagebot now training. Please choose a side to sort after putting down an object')
#xdisplay_image.send_image('path/to/initial/image')
def drawProductionUI(self):
#TODO#
pass
print('Garbagebot finished training. Please put down an object and let the robot do the sorting')
......@@ -50,15 +50,11 @@ def isolate_shape(img, rect=None, only_shape=True):
def name_to_features(name):
color = {'RED': 0, 'BLUE': 1, 'GREEN': 2}
shape = {'CUBE': 0, 'SPHERE': 1, 'TRIANGLE': 2}
# features = np.zeros(len(color.keys())*len(shape.keys()))
colors = np.zeros(len(color.keys()))
shapes = np.zeros(len(shape.keys()))
color = [v for k, v in color.items() if name.startswith(k)][0]
shape = [v for k, v in shape.items() if name.endswith(k)][0]
# features[color*3+shape] = 1
# return features
# print(str(color) + ' ' + str(shape) + ': ' + str(features))
colors[color] = 1
shapes[shape] = 1
return np.append(colors, shapes)
......@@ -67,10 +63,8 @@ def out_to_features(out):
color = ['RED', 'BLUE', 'GREEN']
shape = ['CUBE', 'SPHERE', 'TRIANGLE']
# outputs ranked according to probabilities,
# take the most probable one
# out = out[0]
# c_i = out/3
# s_i = out%3
# first 3 are probabilities for color,
# next 3 are probabilities for shape
c_i = np.argmax(out[3:])
s_i = np.argmax(out[:3])
return {'color': color[c_i], 'shape': shape[s_i]}
......@@ -110,11 +104,13 @@ def augment(x, y, nb=20):
return np.append(x, np.array(x_aug), axis=0), np.append(y, np.array(y_aug), axis=0)
def preprocess_sample(img, crop_x=(210,430), crop_y=(0, 200)):
# remove background
img = isolate_shape(img, only_shape=False)
# crop to focus on item
img = img[crop_y[0]:crop_y[1], crop_x[0]:crop_x[1], :]
# downscale image
img = cv2.resize(img, (0,0), fx=0.5, fy=0.5)
# img = cv2.GaussianBlur(img,(5,5),0)
cv2.imwrite('test.jpg', img)
# normalise image
img = img/255.
return img
......@@ -130,9 +126,12 @@ def preprocess(data):
return x, y
def create_model(x_shape=[None, 100, 110, 3], out=9, path='model/model.m'):
# input is the colored image, this will be spread over 2 nets, one for each feature
color = tflearn.input_data(shape=x_shape)
# the shape-learning net will change the img to grayscale
gray = tflearn.custom_layer(color, tf.image.rgb_to_grayscale)
final = []
# make the same net for each feature, using conv nets
for name, net in {'color': color, 'shape': gray}.items():
net = tflearn.conv_2d(net, 32, (3, 3), activation='relu')
net = tflearn.max_pool_2d(net, (2,2))
......@@ -147,6 +146,7 @@ def create_model(x_shape=[None, 100, 110, 3], out=9, path='model/model.m'):
net = tflearn.regression(net, optimizer='adam', loss='categorical_crossentropy')
final.append(net)
# merge both loss functions into one
final = tflearn.merge(final, 'concat')
return tflearn.DNN(final, tensorboard_verbose=0) #, checkpoint_path=path)
......@@ -169,6 +169,11 @@ def load_model(path, x_shape, out):
model.load(path)
return model
# this file can be used as main in two ways:
# 1. train the network, needs path to dataset as arg
# 2. use a trained network to predict an image.
# 1st arg: path to image to classify
# 2nd arg: path to model weights
if __name__ == '__main__':
path = '.'
pred_path = None
......
......@@ -22,18 +22,15 @@ def run(baxter, learner, scene=None):
baxter.left.move(baxter.left.top)
rospy.sleep(1)
print('gonna take snap')
img = baxter.left.take_snapshot()
print('took snap')
cv2.imwrite('test.jpg', img)
# cv2.imwrite('test.jpg', img)
p = predict(learner, img)
print(p)
def predict(learner, img):
cv2.imwrite('test.jpg', img)
# cv2.imwrite('test.jpg', img)
img = pred.preprocess_sample(img)
pred = learner.predict_label([img])[0]
return pred.out_to_features(pred)
return pred.predict_features(learner, img)
if __name__ == '__main__':
moveit_commander.roscpp_initialize(sys.argv)
......@@ -56,5 +53,5 @@ if __name__ == '__main__':
b.both.move(b.both.neutral)
rospy.sleep(1)
b.left.take_snapshot()
learner = pred.load_model(model_path, [None, 100, 110, 3], 9)
learner = pred.load_model(model_path, [None, 100, 110, 3], 3)
run(b, learner, scene)
import numpy as np
import cv2
import matplotlib.pyplot as plt
def getSobel(channel):
sobelx = cv2.Sobel(
channel, cv2.CV_16S, 1, 0, borderType=cv2.BORDER_REPLICATE)
sobely = cv2.Sobel(
channel, cv2.CV_16S, 0, 1, borderType=cv2.BORDER_REPLICATE)
sobel = np.hypot(sobelx, sobely)
return sobel
def findSignificantContours(img, sobel_8u):
image, contours, heirarchy = cv2.findContours(sobel_8u, cv2.RETR_TREE,
cv2.CHAIN_APPROX_SIMPLE)
# Find level 1 contours
level1 = []
for i, tupl in enumerate(heirarchy[0]):
# Each array is in format (Next, Prev, First child, Parent)
# Filter the ones without parent
if tupl[3] == -1:
tupl = np.insert(tupl, 0, [i])
level1.append(tupl)
# From among them, find the contours with large surface area.
significant = []
tooSmall = sobel_8u.size * 5 / 100 # If contour isn't covering 5% of total area of image then it probably is too small
for tupl in level1:
contour = contours[tupl[0]]
area = cv2.contourArea(contour)
if area > tooSmall:
cv2.drawContours(
img, [contour], 0, (0, 255, 0), 2, cv2.LINE_AA, maxLevel=1)
significant.append([contour, area])
significant.sort(key=lambda x: x[1])
return [x[0] for x in significant]
def segment(img):
blurred = cv2.GaussianBlur(img, (5, 5), 0) # Remove noise
# Edge operator
sobel = np.max(
np.array([
getSobel(blurred[:, :, 0]),
getSobel(blurred[:, :, 1]),
getSobel(blurred[:, :, 2])
]),
axis=0)
# Noise reduction trick, from http://sourceforge.net/p/octave/image/ci/default/tree/inst/edge.m#l182
mean = np.median(sobel)
# Zero any values less than mean. This reduces a lot of noise.
sobel[sobel <= mean] = 0
sobel[sobel > 255] = 255
cv2.imwrite('output/edge.png', sobel)
sobel_8u = np.asarray(sobel, np.uint8)
# Find contours
significant = findSignificantContours(img, sobel_8u)
# Mask
mask = sobel.copy()
mask[mask > 0] = 0
cv2.fillPoly(mask, significant, 255)
# Invert mask
mask = np.logical_not(mask)
# Finally remove the background
img[mask] = 0
return img
# fname = path.split('/')[-1]
# cv2.imshow('output', img)
# cv2.key
# cv2.imwrite('output/' + fname, img)
# print (path)
# segment('original-small.jpg')
def isolate_shape(img, rect=None, only_shape=True):
img = cv2.GaussianBlur(img, (5, 5), 0)
# plt.imshow(img)
# plt.show(block=False)
height, width = img.shape[:2]
# Create a mask holder
mask = np.zeros(img.shape[:2], np.uint8)
# Grab Cut the object
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)
# "Hard Coding the Rect The object must lie within this rect."
if not rect:
rect = (300, 0, 100, height / 2)
cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 2, cv2.GC_INIT_WITH_RECT)
# grabCut
mask = np.where((mask == 2) | (mask == 0), 0, 1).astype('uint8')
img1 = img * mask[:, :, np.newaxis]
# Get the background
background = img - img1
# Change all pixels in the background that are not black to white
background[np.where((background > [0, 0, 0]).all(axis=2))] = [
255, 255, 255
]
# Add the background and the image
final = background + img1
if only_shape:
return background
return final
# Load the Image
# img = cv2.imread(
# '/home/fabio/dev/workspaces/personal/ctai/GarbageBot/src/detecting/dataset/REDTRIANGLE.right.1.jpg'
# )
# plt.imshow(isolate_shape(img, rect=(300, 0, 100, 300), only_shape=False))
# plt.show()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment