T.C. MARMARA UNIVERSITY FACULTY of ENGINEERING COMPUTER ENGINEERING DEPARTMENT

T.C. MARMARA UNIVERSITY FACULTY of ENGINEERING COMPUTER ENGINEERING DEPARTMENT CSE497 Engineering Project Project Specification Document INTELLIGENT WALL CONSTRUCTION BY MEANS OF A ROBOTIC ARM Group Members 150112072 Filiz DALIP 150112922 Moustafa MEMET Supervised by Asst. Prof. Peter Schüller Co-Advisor Assoc. Prof. Borahan TÜMER

1. Problem Statement Nowadays arm robots are used almost everywhere like welding, material handling, painting and assembling products. When we look at these, we can spot a huge gap in construction sector. In the very near future we might need the help of some arm robot to construct us small buildings or miniature houses using wall materials like bricks. We study this topic to build skills in this important area. 2. Problem Description and Motivation Description Robotic arms have become the keystones of industrial production facilities. Robots are widely used in heavy industrial cutting and automotive industry. However we noticed that the robotic arms are still not well developed in construction sector. With this project we are going to realize construction automation and it s going to be a major step in creating huge construction facilities. We will create both hardware and software for autonomous wall building. Because autonomy is important, so that humans do not need to specify each small movement of the robot. The working principle of our robotic arm is like this: The arm will reach the location where the lego bricks are given as input by moving on the conveyor belt. As soon as it reaches there system will measure the size and the dimensions of the input brick. Then the gripper will grab the piece and it will place it to best available position. This way, by minimizing the supply usage we optimize the construction. Motivation Robotic arms which are used in industrial sectors becoming brick constructing and wall building robots is our biggest motivation. Along with that, by using image processing technology, building a wall with lego pieces and lego statues with robotic arms can have a remarkable contribution to lego sector. Also with today s technology we can make anything we want with unlimited Lego pieces using Lego Digital Designer. On top of that they are making games and films with Lego for different platforms. 1

3. Aims of the Project Construct a jointed arm robot by using Lego pieces which is similar to Figure 1. Move the robotic arm right, left, up and down stably. Be able to hold the lego piece with the gripper. Mount the robotic arm to conveyor belt (Figure 2), and remove the bottom rotation motor. Stabilize the robotic arm and conveyor belt so that the vibration minimizes. Place a webcam onto gripper. Perceive the size and dimensions of a lego piece using image processing with a webcam. Compute the best available position to place the lego piece on the wall. Place the lego piece that we picked up in our grip to the best available position. Build a wall with the lego pieces by placing them into right positions. The main purpose of this robot will be to build a wall by using Lego pieces autonomously. Figure 1: Example Jointed Arm Robot Figure 2: Conveyor Belt Assembly 2

4. Related Work The International Organization for Standardization (ISO) gives a definition of an industrial robot like this: "an automatically controlled, reprogrammable, multipurpose, manipulator programmable in three or more axes, which may be either fixed in place or mobile for use in industrial automation applications." Quoted from Wikipedia. See [1] for more. What we are going is almost the same with this definition of industrial robot. It will be indeed reprogrammable and automatically controlled but it will be working stationary in 2 axes. Industrial robots usually consist of a jointed arm (multi-linked manipulator) and an end effector that is attached to a fixed surface. One of the most common type of end effector is a gripper assembly. Quoted from Wikipedia. See [1] for more. As described in the quote above we are going to make the most common type of end effector. Our gripper will be used to hold the lego pieces and release them to correct positions. There are different types of industrial robots: Cartesian Robot: Used for pick and place work, application of sealant, assembly operations, handling machine tools and arc welding. It s a robot whose arm has three prismatic joints, whose axes are coincident with a Cartesian coordinator. Cylindrical Robot: Used for assembly operations, handling at machine tools, spot welding, and handling at diecasting machines. It s a robot whose axes form a cylindrical coordinate system. Spherical Robot: Used for handling machine tools, spot welding, diecasting, fettling machines, gas welding and arc welding. It s a robot whose axes form a polar coordinate system. Scara Robot: Used for pick and place work, application of sealant, assembly operations and handling machine tools. This robot features two parallel rotary joints to provide compliance in plane. Jointed Arm Robot: Used for assembly operations, diecasting, fettling machine, gas welding, arc welding and spray painting. It s a robot whose arm has at least three rotary joints. Quoted from Wikipedia. See [2] for more. We are going to build our robot on a conveyor belt which will allow us to move it in x-axis. The arm part which will be placed on top of the conveyor belt will make movements in y-axis. Since its moving in only 2 axis, our robot will belong to cartesian robot category. 3

Minimization of human power, increasing the production by reducing the error margin, makes a great improvement for today s technology. Along with that the robotic arm which was created with high expenses produces the desired product flawlessly by assembling the lego pieces together. You can find the article about this robotic arm in references section [3]. Another project accomplishes almost the same mission as the mega robot above. This one is created using LEGO Mindstorms NXT kit [4] and its working principle is slightly different than the other one. It s created using the 3D printer logic. The desired piece is placed to correct place stably by using 3 motors. Design and the related video can be found in references section [5]. 5. Scope of the Project Our scope begins with the well assembly of the robotic arm and the conveyor belt. We used the Mindstorms EV3 45544 set [6] to build a cartesian robot. After making some motor and durability tests we made some changes on the original arm. By changing some bricks we rotated the gripper part of the arm to suit our needs. We are going to integrate a conveyor belt which is made with LEGO Technic 42028 Set. This belt will be placed underneath robotic arm to move it linearly to right and left. As soon as we arrange the arm system onto belt we are going to mount a small webcam onto gripper to read the image of the lego pieces which are given as input. Here we are going to analyze the image using MATLAB s image processing functions. We are going to measure the dimensions and the size of the lego piece and use an algorithm to find the best possible position to place the brick. Using these combined we are going to build a single wall which is made from lego bricks. After successfully building a wall if there is time left to make a house using these walls. 4

6. Success Factors and Benefits We can conclude that our project has become successful only if the following statements are satisfied: Robotic arm grabbing the lego piece from their place Robotic arm moving freely and flawlessly on the conveyor belt Understanding the grabbed piece correctly using image processing Placing the lego piece to correct place by using the correct algorithms and calculations After achieving the above we are going to arrange the software of the robotic arm to build us a wall autonomously. We are going to make an input location for lego bricks on the table. Robot will always come back to that location to obtain new bricks. According to given bricks the robot will place them to the best available position. The only human interaction in this process is going to be giving the input bricks. The benefits of the project can be listed as follows: The most important benefit might be the reduction of the workload for the workers because of its use in construction industry. If the application of the project is done correctly it might provide a great profit to the owner. Because robotizing one job improves the quality of products and reduces the expenses drastically. It can make contribution to economy and technology can be sold to foreign countries. The gap in the construction sector might be closed with the autonomous builders like this robotic arm. With the replacement of the gripper part, the arm can also be used to spill cement between the bricks. Along with these a complete schematic of the total project can be taught to robot. Getting this done can give you a complete building with the help of worker robots. 5

7. Methodology and Technical Approach Figure 3: Flowchart of the tasks Our project consists of three parts. These are hardware and robot assembly, robot movement algorithms and software and image processing software. The things that we need for the robot assembly: Lego Mindstorm EV3 Set 45544 [6] Robot Arm Programmable Brick Motors Touch Sensor Conveyor Belt Camera Lego Pieces Lego Mindstorm EV3 Set 45544 [6]: This set consists of all the following parts that we need for our project. Robot Arm: Using this set we can make number of different designs. What we are going to build is the robotic arm with some modifications. 6

Programmable Brick: The EV3 Brick serves as the control center and power station for your robot. Large Motor: Lets you program precise and powerful robotic action. Medium Motor: Maintains precision, while trading some power for compact size and faster responses. Touch Sensor: Makes your robot respond to touch, recognizes three conditions: touched, bumped and released. Conveyor Belt: We will use Lego Technic 42028 Sets to make suitable conveyor belt to mount the robot arm. Camera: A small webcam will be used to read the dimensions and the size of lego bricks. This webcam will be mounted onto gripper. Lego Pieces: We are going to need a lot of pieces to make tests. The things that we need for the robot movement software: MATLAB [7] LEGO Mindstorms EV3 library for MATLAB [8] MathWorks Image Processing Toolbox [9] MATLAB [7]: MATLAB (matrix laboratory) is a multi-paradigm numerical computing environment and fourth-generation programming language. A proprietary programming language developed by MathWorks, MATLAB allows matrix manipulations, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs written in other languages, including C, C++, C#, Java, Fortran and Python. LEGO Mindstorms EV3 library for MATLAB [8]: Mathworks provides free libraries for LEGO Mindstorms EV3 robot control. We are going to use these libraries to move our robot. MathWorks Image Processing Toolbox [9]: This toolbox consists many image processing functions. We are going to use these libraries to understand the dimensions of the lego pieces that we grab. 7

Image processing software and algorithms: Image Processing is processing of images using mathematical operations by using any form of signal processing for which the input is an image, a series of images, or a video, such as a photograph or video frame; the output of image processing may be either an image or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it. Images are also processed as three-dimensional signals where the third-dimension being time or the z-axis. Quoted from Wikipedia. See [10] for more. In our case, the images that we are going to process are going to be photographs which are taken by our webcam. These images will be 3-dimensional signals, each representing a color space (Red, Green, Blue). These will be analyzed with the help of MATLAB s image processing toolbox. We are going to integrate robot movement and image processing together using the MATLAB software. MATLAB can support the Mindstorms EV3 controls using its add-on extensions [8]. We will connect the EV3 brick to our computer along with the webcam that we are going to mount onto gripper. Webcam will receive the image and convert it to JPEG format. We will analyze these images using MATLAB and understand the dimensions of the given lego piece. Mathworks also provides a huge library [9] for image analysis and segmentation. Using these libraries we are going to develop our software on MATLAB. 8

8. Professional Considerations Methodological Considerations will realize. In the figure below you can find the block diagram of the process that our robotic arm Figure 4: Block Diagram of the process 9

We put the input bricks to predefined input place consecutively. Our gripper will take the piece from there and move it in front of the camera. The camera will capture the picture of the lego brick and will try to understand the size of it. Then it will run some algorithms to compute the best available position for the new brick. After that we will move the robot to proper position and leave the brick to best available position. Repeating this process builds a wall. Societal and Ethical Considerations Societal and ethical considerations has been taken into consideration while preparing this document. One social impact could be fewer building jobs. Since the robotic arms are going to take the place of building workers. This also might increase the need of the programmers for the robots. Economically this project will help to reduce the human power on some specific jobs. It s going to increase the efficiency and improve the construction process by reducing the time required to complete it. The realization of this project definitely has no physical and biological impacts. Along with these, this project will be designed with a good care of environmental conditions. There will be no harm to natural environment. Legal Considerations The usage of MATLAB and its libraries in our project may need licenses and usage permissions if this project ever comes to market. Please note that we do not apply the project in reality and the robotic arm that we are going to build is small so it s not dangerous. If it was a real size robot, it would need ethical, work safety, engineering and many more permissions. 10

9. Management Plan The steps to take and the deadlines to follow in two terms is as follows: 1. Literature research, reading the reviews of the related work and finding ideas to apply them to our project. (completed) 2. Arrangement of required equipments and additional parts. (completed) 3. Assembly of the parts for prototype. (completed) 4. Preparation of the Project Specification Document. (in progress) 5. Placement of the robotic arm onto conveyor belt. Redesign of prototype. 6. Running tests for motors, sensors and Brick controller. Hardware stability control. 7. MATLAB movement tests. 8. Achieving the movement in 3 axis. 9. Grabbing the lego pieces with the gripper and placing them into positions manually. 10. Webcam installation and and capturing the images. 11. Processing of the images, storing the brick sizes and dimensions. 12. Integration of image processing and robot movement. 13. Development and improvement of algorithms. 14. Placement of the sequentially given bricks to best available position. 15. Running tests with different brick sizes. Division of responsibilities among teammates Filiz Dalıp Tasks: 6, 7, 8, 9, 11, 12, 13, 14, 15 Moustafa Memet Tasks: 5, 8, 9, 10, 11, 12, 13, 14, 15 We have successfully completed up to 4th task. Within the following weeks we are going to complete the hardware design of the project. Up to 6th phase we are working on our hardware design. We will gently try to make our arm more stable that the original. Because the precision is going to be really important in the following phases. Running some stability tests may take some longer time. After completing the hardware, we are going to run some movement tests with the help of the software. Here we will try and see our design s capabilities. We are going to move the robot manually and achieve the movement in 3 axis. All possible movements that we are going to need later will be realized in phases 7-9. 11

Once we can be sure that our robot has full movement capabilities in 3 axis, we are going to install a camera and start working on image processing algorithms. Our aim at first is to be able to read the brick correctly. As soon as we accomplish reading the images we are going to work on placement algorithms. We are going to keep the current state of the bricks in our memory and find the best possible place for the new brick. Until the end of the April we will be working on these algorithms. Below you can see the timeline with milestones: Figure 5: Gantt chart for timeline with milestones Risk Management One of the major risks can be the lack of stabilization on our system. We might not be able to build an arm from legos with full precision. In this case we are going to try our best to make them stable and strong for our job. Another risk could be the compatibility of EV3 software with MATLAB. If they are not fully compatible, we might need to try another way to control the motors. Reading the bricks from one perspective may not be enough to understand the size and dimensions of bricks. In this case we might reduce the amount of possible bricks we use. 12

References [1]- Wikipedia, Robots [Online] Available: https://en.wikipedia.org/wiki/robot (Date of Access: 24 October 2016) [2]- Wikipedia, Robotic Arms [Online] Available: https://en.wikipedia.org/wiki/robotic_arm (Date of Access: 24 October 2016) [3]- Singularityhub, Motoman robot impresses with lego building prowess (by Aaron SAENZ) on Jan 08,2010 [Online] Available: http://singularityhub.com/2010/01/08/motoman-robot-impresses-with-lego-building-prowess-video/ (Date of Access: 25 October 2016) [4]- Wikipedia, Lego Mindstorms NXT [Online] Available: https://en.wikipedia.org/wiki/lego_mindstorms_nxt (Date of Access: 26 October 2016) [5]- Wired,Machine made of lego builds anything you want out of lego (by Priya GANAPATI) on October 20,2010 [Online] Available: https://www.wired.com/2010/10/legobot/ (Date of Access: 26 October 2016) [6]- Lego Education, [Online] Available: https://education.lego.com/en-au/lego-education-product-database/mindstorms-ev3/45544-lego-mindsto rms-education-ev3-core-set (Date of Access: 27 October 2016) [7]- Wikipedia, Matlab [Online] Available: https://en.wikipedia.org/wiki/matlab (Date of Access: 28 October 2016) [8]- Mathworks, Lego Mindstorms EV3 support from Matlab [Online] Available: https://www.mathworks.com/hardware-support/lego-mindstorms-ev3-matlab.html (Date of Access: 29 October 2016) [9]- Mathworks, Perform image processing, analysis, and algorithm development [Online] Available: https://www.mathworks.com/products/image/ (Date of Access: 29 October 2016) [10]- Wikipedia, Image Processing [Online] Available: https://en.wikipedia.org/wiki/image_processing (Date of Access: 30 October 2016) 13