st"> nd"> rd"> cm"> degree"> kg"> px"> s"> unit"> bool"> char"> unsigned char"> short"> unsigned short"> int"> unsigned int"> long"> unsigned long"> long long"> unsigned long long"> half"> float"> double"> long double"> vec"> vec1"> vec2"> vec3"> vec4"> vec1ub"> vec2ub"> vec3ub"> vec4ub"> vec1i"> vec2i"> vec3i"> vec4i"> vec1f"> vec2f"> vec3f"> vec4f"> vec1d"> vec2d"> vec3d"> vec4d"> mat"> mat1"> mat2"> mat3"> mat4"> mat1f"> mat2f"> mat3f"> mat4f"> mat1d"> mat2d"> mat3d"> mat4d"> igl"> igl1"> igl2"> igl3"> igl4"> igl1f"> igl2f"> igl3f"> igl4f"> igl1d"> igl2d"> igl3d"> igl4d"> hom"> hom1"> hom2"> hom3"> hom4"> hom1f"> hom2f"> hom3f"> hom4f"> hom1d"> hom2d"> hom3d"> hom4d"> quatf"> quatd"> rot2d"> rot3d"> rot4d"> Catmull"> Euler"> Gauß"> Hadamard"> Hermite"> Hesse"> Jacobi"> Jacobian"> Newton"> Nyquist"> Rom"> SDV"> WETA"> Apple"> Adobe"> Autodesk"> DuckDuckGo"> Google"> Microsoft"> SideFX"> The Foundry"> After Effects"> Houdini"> Nuke"> Flame"> Maya"> cairo"> Doxygen"> Sphinx"> Graphviz"> Highlight"> jQuery"> jQueryUI"> OpenOffice"> CDML"> OpenGL"> GLUT"> freeglut"> LaTeX"> subversion"> MySQL"> Php"> valgrind"> cluster-ssh"> Ubuntu"> CentOS"> 3DE4"> 3DE4r4b1"> 3DE4r4b2"> 3DE4r4b3"> 3DE4r4b4"> 3DE4r4"> 3DE4r6b1"> 3DE4r6b2"> 3DE4r6b3"> 3DE4r6"> tde4mhpd"> 3DE4/MHP"> tde4/widget-tk"> flcd"> flcd5"> MHP"> VL"> flcd"> flcd5"> CDB"> Main Window"> Overview Controls"> Manual Tracking Controls"> Distortion Grid Controls"> Autotracking Controls"> Lineup Controls"> 3D Orientation Controls"> F1"> F2"> F3"> F4"> F5"> F6"> Ctrl"> Shift"> Alt"> 3DE4"> Windows"> Edit"> Calc"> Python"> Adjustment"> File"> Export"> Object Browser"> Deviation Browser"> Timeline Editor"> Curve Editor"> Attribute Editor"> Parameter Adjustment Window"> Advanced Options"> Network Calc"> Image Controls Window"> Python Console Window"> Online Help Window"> Preferences"> Network Calculation"> Network Calculation Mode"> Enabled - Static Master Node"> Enabled - Dynamic Self Organizing Network"> Off"> Master Node Hostname"> Allow For Network Calculation"> Frame Range Calculation"> Run Python Script..."> Rescan Python Directories"> Open Python Console..."> Project"> Camera"> Camera"> Weight"> Live Action Footage"> First Frame is Frame"> Lens"> Lens..."> Focal Length"> Use From Lens"> Static"> Dynamic"> Focus Distance"> Rolling Shutter Compensation"> Synchronization"> Stereoscopic"> Orientation"> Lefthand Camera"> Righthand Camera"> Timeshift"> Point Group"> Point"> Point"> Position XYZ"> 2D Tracking"> 3D Calculation"> Survey Type"> Survey Free"> Approximately Surveyed"> Exactly Surveyed"> Lineup Only Surveyed"> Approx. Survey Radius"> Point Weighting"> Calc Automatically"> 3D Model"> Lens"> Lens Distortion"> Dynamic Lens Distortion"> Scene"> Cameras"> Point Groups"> Constraints"> Lenses"> blob"> core"> gui script"> gui scripts"> modifier script"> modifier scripts"> parent"> responder"> computational"> news"> updater_udp"> scanner"> deputy"> self"> complex"> len"> __repr__"> __str__"> +"> −"> *"> /"> +="> −="> *="> /="> dotsq"> dot"> tensq"> ten"> wdg"> had"> norm1"> norm2"> norminf"> unit"> para"> ortho"> dual"> invert"> conjugate"> quat"> axis"> angle"> angles"> trans"> trace"> sub"> adj"> det"> invert"> mat"> 3DE4"> ||"> a"> b"> p"> u"> pl"> ul"> ml"> pr"> ur"> mr"> p' r"> cl,x"> cl,y"> cl,z"> cr,x"> cr,y"> cr,z"> c' r,z"> ccenter"> ex"> ey"> ez"> ddepth"> dvert"> dioc"> pconv"> 1"> R3"> &u;"> &u_l;"> &u_r;"> stereo"> center"> Attribute Editor"> Cameras"> Stereoscopic"> Rotation Policy"> rotation policy"> rotation policies"> Allow y-Rotation Only"> Allow Rotation Around all Axes"> parallaxe"> interocular"> Interocular"> Curve Editor"> vertical shift"> Vertical Shift"> depth shift"> Calc"> Zoom Curve"> Calc Zoom Curve From Scratch"> Finetune Zoom Curve From Scratch"> w"> wa"> wb"> wr"> h"> ha"> hb"> hr"> gles::math"> ldpk"> transwarp"> minifl"> img"> img_memptr"> int"> true"> false"> float"> n"> selector"> segment"> collect/apply"> ]> 3DE4/MHP
&tde4mhp; Manual Network Calculation Science-D-Visions, 2018-11-12 Contents 1About &tde4mhp;
1.1Basic principle
1.2Versions
2Quick guide for &tde4; users
2.1Preferences
2.1.1&linux;, &osx;
2.1.2Windows
2.2Parameter Adjustment
2.3Debugging / Support
3Quick guide for administrators
3.1Static network
3.1.1Adding more nodes to the network
3.2Automatic network
3.2.1Adding more nodes to the network
4Reference manual
4.1The excutable &tde4mhpd;
4.1.1Starting the node
4.1.2Terminating the node
4.1.3Command line options, environment variables, configuration file
4.1.4Important options
4.1.5Less important options
4.1.5.1Configuration file
4.1.6Summary
4.2Requirements and recommendations
4.2.1The owner of tde4mhpd
4.2.2Environment variables $PATH and $HOME
4.2.3Shell commands
4.2.4Default route
4.2.5TCP, UDP, Ports, Firewall
4.2.5.1¢os;
4.2.5.2&ubuntu;
4.3Limitations
4.4The html-interface
4.4.1The page hosts.html
4.4.2The page properties.html
4.4.3The pages processes.html and processes_network.html
4.4.4The page load.html
4.5Lens distortion plugins
5Miscellaneous
5.1Todo
5.2An incomplete list of useful shell commands
5.3Multi-host administration
1About &tde4mhp; &tde4mhp; stands for 3DE4 Multi Host Processing. It is a program which allows &tde4; to do parameter adjustment on many hosts instead of a single host. Parameter adjustment in &tde4; can take a long time, when many parameters have to be determined in a brute-force method. On a single host, this means a &tde4;-project has to be processed over and over again with modified parameters. This is usually done by multiple threads. In organizations with many seats, machines are often not used to their capacity, because for many operations only one CPU core is needed, while the remaining ones do little or nothing. These free capacities are utilized by &tde4mhp; for parameter adjustment. Depending on the number of available machines this leads to a considerable performance gain. This manual describes how to set up and use &tde4mhp;. 1.1Basic principle In practice, &tde4mhp; is an executable &tde4mhpd; which is either launched and controlled by means of &tde4;'s GUI, or it runs a standalone program. A running instance of &tde4mhpd; is called a node. Once started, the node tries to find other nodes in the local area network and form a network on the application layer, which we call the &mhp;-network. The nodes in this network communicate with each other over TCP/UDP. Each node also provides an http-interface, so that it can be inspected and modified by the administrator or &tde4; user by means of a common webbrowser. 1.2Versions The version number of &tde4mhpd; has the form
a.b.c
c is increased in order to indicate bug fixes and minor changes. b indicates changes in the underlying core technology of &tde4;. Nodes with same major version a but different subversion number b can be managed by the same chief. However, &tde4; will only send jobs to nodes with matching major version and matching subversion number. A change in a indicates a severe change in protocols, which requires that each node is upgraded. We will try to avoid this. In this section we keep track of the development of &tde4mhpd;.
Version&tde4; Rel.ReleasedChanges
1.19.0&tde4r6b2;Bugfixes in &tde4;'s core
1.18.0&tde4r6b1;Adaptive load balancing, better handling of small projects
...Changes in &tde4; retraced
1.5.0Bugfixes in &tde4;'s core
1.4.0Bugfixes in &tde4;'s core
1.3.0Lens models which depend on focal length and focus distance; point constraints
1.2.0Bugfix: Sync + Lock Channels
1.1.0&tde4r4b2;2014-05-09 Bugfix: &tde4mhpd; more robust when host resolution is blocking. Bugfixes in &tde4;'s core, most of them related to stereo and sync projects.
1.0.69&tde4;R4b12014-04-02First release
Versions of this document:
VersionReleasedChanges
52018-11-07firewall commands under &linux;, bugfixes, appearance
4unpublished multi-admin tool mentioned
3unpublishedminor changes in design
22014-05-09minor changes, typos
12014-04-09first release
2Quick guide for &tde4; users 2.1Preferences 2.1.1&linux;, &osx; Network calculations are enabled in &tde4;'s preferences. If they are enabled, a &tde4mhpd; node will run on your machine. There are two ways of connecting your node to nodes on other machines:
  • &item_enabled_static_master_node; - In this mode you enter the name or ip-address of the chief node in the text field &entry_master_node_hostname;. If your organization is using dedicated hosts for network calculations, the system administrator will give you the details.
  • &item_enabled_dynamic_self_organizing_network; - In this mode your node will try to find other nodes by scanning the local area network. In the ideal case this will take a few seconds, but it could also take much longer. We have put some effort into finding other nodes without burdening your network, so we think it's worth a try.
By the slider &entry_allow_for_network_calculation; you specify how many of your CPU cores are provided to the community so that they can work on jobs submitted by other users in the &mhp;-network. The remaining ones are reserved for jobs initiated by yourself. It's not easy to give a reasonable default value for this. On machines with more than 4 CPU cores we recommend to keep at least 4 CPU cores private. The preferences allow to specify up to three address ranges to scan for other nodes. The primary ip address range should already contain reasonable default values, namely the netmask of the local area network your host belongs to. Typical default values here are:
  • 10.0.0.0, 8 bit
  • 172.16.0.0 to 172.31.0.0, 12 bit
  • 192.168.0.0, 16 bit
  • 192.168.0.0 to 192.168.255.0, 24 bit
Generally, there should not be reason to modify these values.
2.1.2
Windows In the current release of &tde4mhp; there is no &tde4mhpd; node running on &windows;. Nonetheless, &tde4; users working on &windows; can connect to existing &mhp;-networks. If your company decides to use &tde4mhp; your system administrator should set up a static &mhp;-network on &linux; and/or &osx; machines. Once this network is up and running, you only need to enter the name or ip-address of any of the hosts participating in the network in the text field &entry_master_node_hostname; in &tde4;'s preferences: For calculations, &tde4; will use all community threads available in the &mhp;-network plus all CPU cores of the local host you are working on. 2.2Parameter Adjustment The main purpose of &tde4mhp; is to accelerate parameter adjustment. In the ¶meter_adjustment_window; you see a small section named &entry_network_calc; underneath the parameter entries. The toggle button enables or disables the network calculation. If it is turned off, other users will still be able to use the common CPU cores on your machine. If it is turned on, any parameter adjustment you start will be submitted to the &mhp;-network. The numbers (here "0/57 [11 hosts]") indicate how many threads are working on your job (here: 0) and how many threads there are at your service (here: 57), provided by 11 hosts. The second number is the number of public threads / CPU cores plus your private threads / CPU cores. The green bar shows that someone else is currently doing calculations in the network. &tde4mhp; is clearly focused on brute-force calculations. For brute-force adjustment, network calculations are accelerated considerably, although we do not have reliable numbers on the performance scaling behaviour. For adaptive adjustment, network calculations are possible, but they are accelerated only between 10% to 70%, sometimes 100%, much less than brute-force adjustments. 2.3Debugging / Support The most critical part of &tde4mhp; on the client side, i.e. in connection with &tde4; is the startup and shutdown of the node and connecting it to other nodes. &tde4; examines the environment variable TDE4MHP_CLIENT_LOG and writes debugging data to the file specified there. If you send us a file with these data, this will help us a lot getting it running. If you are curious, set this to /dev/tty on &linux;/&osx; or to CON on &windows;... 3Quick guide for administrators First of all, there should be as little work as possible for the administrator. The entire distributed computing business is useless if all resources (i.e. processing time) saved in one part of the company need to be invested somewhere else (administration time). But if work needs to be done, there should be appropriate tools. For this reason, we have equipped the nodes with a simple webserver functionality, so that the current state of each node or the entire &mhp;-network can be examined and modified in a common webbrowser. Most of the time during development, we have been using Firefox, but Chrome, Opera or Safari should do as well. The browser interface is explained in detail in section 4.4.

Although this is meant to be the quick guide, we will have to define a few things. When the user starts &tde4; with network calculations enabled, a &tde4mhpd; node will start as well. This node is controlled via &tde4;'s GUI, and therefore we call it a controlled node. On the other hand, it might be interesting to increase computational power by adding more machines without running &tde4;. Nodes running on hosts without &tde4; are controlled over the brwoser interface or by means of scripts. We will call these nodes standalone nodes in the following.

An MHP-network is a distributed system with a central node, which we call the chief. The other nodes are called loyals. Messages are passed from chief to loyals and vice versa. The chief collects reports from all nodes and communicates with clients (&tde4;). The role of the chief is not fixed within the &mhp;-network; when the node is shut down regularly it moves on to another node, or you as the administrator can shift it manually from one node to another.

In the following we distinguish between static network and automatic network. Although it may sound like completely different modes of operation, this is not the case. The only difference is the way the nodes find each other in order to form the &mhp;-network. Once the network is up and running, there is no such thing like a static or automatic network.

3.1Static network A static network makes sense if you have at least one standalone node. You start this node as the chief and advise all &tde4; operators to set their &entry_network_calculation_mode; to &item_enabled_static_master_node; and to enter the ip-address of your static chief in the text field &entry_master_node_hostname;. All nodes controlled by &tde4; will then directly connect to the chief node. Figure 1: A typical network with controlled and standalone nodes

Let us have a look at this setup in detail. Assume we have picked a machine name hellboy with ip-address 100.1.1.34 (The address space we use is non-standard for historical reasons...). We are going to start the node now. First, for convenience, we move to the directory the &tde4mhpd; executable is located in.

uwe@hellboy /home/uwe> cd /server/software/linux64/tde4mhpd/
uwe@hellboy software/linux64/tde4mhpd>
We start the node as a daemon and get a status report:
uwe@hellboy software/linux64/tde4mhpd> ./tde4mhpd 
+---------------------------------------------------------------------------+
|                              tde4mhp config                               |
+-------------------+----------------------------------------+--------------+
| name              | value                                  | def'd by     |
+-------------------+----------------------------------------+--------------+
| version           | 1.0.0                                  |              |
| process_mode      | daemon                                 | default      |
| launched_by       |                                        | default      |
| owned_by          | uwe                                    | default      |
+-------------------+----------------------------------------+--------------+
| host              | 100.1.1.34                             | system       |
| port              | 57431                                  | default      |
| chief_host        | 100.1.1.34                             | default      |
| chief_port        | 57431                                  | default      |
| network_mode      | explicit                               | default      |
| subnet_a          | 100.1.0.0/16                           | default      |
| fileserver_base   |                                        | undefined    |
| known_hosts       |                                        | undefined    |
| flcd_host         |                                        | undefined    |
| flcd_port         | 57423                                  | default      |
+-------------------+----------------------------------------+--------------+
| num_cores         | 8                                      | system       |
| num_threads_comm  | 8                                      | default      |
| num_threads_priv  | 0                                      | default      |
+-------------------+----------------------------------------+--------------+
| install_dir       | /guul/server/software/linux64/tde4mhpd | installation |
| plugin_dir        | <install_dir> /                        | undefined    |
| tmp_dir           | /tmp/                                  | default      |
| home_dir          | /home/uwe/.3dequalizer/                | default      |
| config_file       |                                        | default      |
| logfile           | /tmp/tde4mhp.u753.log                  | default      |
| loglevel          | LOG_NOTICE                             | default      |
+-------------------+----------------------------------------+--------------+
| executable path   | ./tde4mhpd                             | installation |
| full logfile path | /tmp/tde4mhp.u753.log                  |              |
+-------------------+----------------------------------------+--------------+
uwe@hellboy software/linux64/tde4mhpd> 
Since there is no error message the daemon should be running now. Let's have a look:
uwe@hellboy software/linux64/tde4mhpd> ps -ef | grep tde4mhpd
uwe      10160     1  0 15:16 ?        00:00:00 ./tde4mhpd
uwe      10170 10160  0 15:16 ?        00:00:00 ./tde4mhpd
uwe      10174 10090  0 15:20 pts/1    00:00:00 grep tde4mhpd
In our example there are two processes, with ids 10160 and 10170. The daemon has spawned a child process (10170) which receives messages from other nodes. We can ignore that. If we try to start the daemon once more, we get an error message, because only one daemon is supposed to run at a time.
uwe@hellboy software/linux64/tde4mhpd> ./tde4mhpd
There already seems to be a node running on this host 
with process id 10160. Please remove this process first 
by 'kill 10160' and then try again.
From the daemon's output we see that hellboy has eight CPU cores, and all of them are dedicated to the community. This makes sense for a standalone node. Without any &tde4; user working directly on the machine there is no need for private CPU cores. What we also see is, that the network mode is explicit which means, the node will not try to find other nodes. Our node now forms a simple network, being the chief itself, and other nodes can check in. That's it in principle. It should be mentioned, that the chief has to master more network traffic than other nodes. It should be a machine with high availability, like the license server. 3.1.1
Adding more nodes to the network Once we have a running chief, it is easy to add more standalone nodes. We start a daemon on a machine named smaug and connect it to the chief node on hellboy:
uwe@smaug /home/uwe> cd /server/software/linux64/tde4mhpd/
uwe@smaug software/linux64/tde4mhpd> ./tde4mhpd -chief_host hellboy
Again, the daemon will output a lot of information, but the interesting part is this one:
...
| chief_host        | hellboy                           | cmdline      |
| chief_port        | 57431                             | default      |
...
| num_cores         | 8                                 | system       |
| num_threads_comm  | 8                                 | default      |
| num_threads_priv  | 0                                 | default      |
...
Our network now consists of two machines, with 16 public CPU cores. At this point, although this is still the quick guide, it's a good idea to know about the browser interface. At address 100.1.1.34:57431/hosts.html we see the following now: Starting at this point you can browse through your &mhp;-network, configure and repair it, if necessary. For details please read section
4.4.
3.2Automatic network When the &tde4; users start their program, a &tde4mhpd; node is started as well. The characteristic property of the automatic network is, as the name indicates, that in principle it does not require any other setup. However, in practice it makes sense to check if all nodes are running as they are supposed to. Also, even in an automatic network you as the administrator can still add standalone nodes in order to increase computational power. Let us assume that there are three instances of &tde4; now running on machines with 8 CPU cores each, all have enabled network calculation in their preferences and provide 4 CPU cores to the community. Then, in the ¶meter_adjustment_window; the users should see the following: The display shows 16 CPU cores: 4 from each of the three machines plus 4 private cores from the machine we took this snapshot from. Now let's add some computational power to this network. 3.2.1Adding more nodes to the network The license server FLCD is well-informed about the &tde4; instances. We can benefit from this by passing the FLCD host (doom in the example below) to &tde4mhpd; on the command line. FLCD will send a list of hosts running &tde4;. From this list the node to be started will find out who is chief and connect &tde4mhpd; directly. The port specification could be omitted, since it's the default for FLCD.
uwe@smaug> ./tde4mhpd -network_mode automatic -flcd_host doom -flcd_port 57423
Assuming that smaug offers 8 CPU cores to the community, &tde4;'s ¶meter_adjustment_window; should display now 24 CPU cores on 4 hosts. In the hosts overview of the browser interface we now see the following. Or graphically: In principle you can build up a large network like this, but in that case it might be good idea to bypass the network scanning procedure and run the chief as described in section
3.1 instead.
4Reference manual 4.1The excutable &tde4mhpd; The executable &tde4mhpd; is located in &tde4;'s installation directory at
${3DE4}/bin/tde4mhpd
in the same directory as the &tde4; executable itself. In order to be started as a controlled node by &tde4; it must reside in this directory. For standalone nodes you can move it wherever you like. If you invoke &tde4mhpd; and pass help or -help, the node will only write an explanation text about its options and exit.
4.1.1
Starting the node Let us consider controlled nodes first. When &tde4; is started with network calculations enabled, &tde4mhpd; is started as well. When &tde4; is terminated, the node keeps on running until it is terminated by intervention of the administrator. Usually, a node controlled by &tde4; is only terminated, if network calculations are disabled in &tde4;'s preferences.

The case of standalone nodes is more interesting for the administrator, so we shall discuss this case in the following. We do this by means of examples, so let us assume our current directory contains the executable of &tde4mhpd;. The following command will start &tde4mhp; as a daemon. It will form a monadic network, consisting only of itself (unless other nodes connect to it). This is the common way to start the chief node in a static setup.

uwe@hellboy> ./tde4mhpd
The following will start the node in foreground mode. This could be interesting for testing and debugging purposes. When started this way, the node can be terminated by &ctrl; C. The process mode can be combined with any other option, so we won't list all the combinations here.
uwe@smaug> ./tde4mhpd -process_mode foreground
The following comand will start the node as daemon and connect it to the chief hellboy. Since 57431 is the default port, the option -port can be omitted here.
uwe@smaug> ./tde4mhpd -chief_host hellboy -chief_port 57431
Assume, you have no idea, on which host the chief is running (and you're not in the mood to find out). Then you can use the automatic network mode and consult the &tde4; license server FLCD (in our case it's called doom). Again, the option -flcd_port could be omitted, since 57423 is the default value for FLCD.
uwe@smaug> ./tde4mhpd -network_mode automatic -flcd_host doom -flcd_port 57423
By the following command, the node will scan the network address range 192.168.32.1 to 192.168.63.254 in order to find the chief or other nodes.
uwe@smaug> ./tde4mhpd -network_mode automatic -subnet_a 192.168.32.0/20
This command will start the node as daemon, read the configuration file and set the thread numbers to 8 and 0, regardless of the values given in the configuration file.
uwe@smaug> ./tde4mhpd -config_file /server/etc/tde4mhp/tde4mhp.conf -num_threads_comm 8 -num_threads_priv 0
4.1.2
Terminating the node There are several ways to terminate a running node. A node is terminated by setting the network mode to &item_off; in &tde4;'s preferences (if it is a controlled node), or you terminate the node via the web interface. In both cases, the &mhp;-network will keep consistent. If you are terminating the chief node, the chief role will be transferred to another node, so that the network keeps consistent and operational. Ideally, this would be transparent for the user of &tde4;. Please have a look at section 4.4.2 for details.

You can also terminate &tde4mhp; from the command line, but in this case the chief role (if your node is chief) is not transferred automatically to another node. Let us assume, your node is not chief. Then a node running in process mode foreground is simply terminated by &ctrl; C. This raises the signal SIGTERM, which allows the node to check out at the chief and clean up whatever needs to be cleaned up. A node running in process mode daemon has not connection to any terminal. At command line you can terminate it by

kill <pid>
which by default sends SIGTERM to the process. A more severe method, which cannot be refused by the process and does not allow the node to clean up is
kill -s KILL <pid>
or equivalently
kill -9 <pid>
This might leave the &mhp;-network in an inconsistent state and should be avoided.

4.1.3Command line options, environment variables, configuration file &tde4mhpd; can be configured in three different ways: by passing options at the command line, by defining environment variables and by configuration file. These three channels can be combined according to your needs. Command line options have the highest priority; any parameter given as environment variable or configuration file is overruled by its command line version. The names of most command line options, environment variables and configuration file option names are related as in the following example:
Environment variableCommand line optionConfiguration file entry
TDE4MHP_NETWORK_MODE-network_modenetwork_mode
There are only a few exceptions: -help, -config_file and -install_dir cannot appear in the configuration file (since that makes little sense). Whenever we talk about an option in the following, we will address it by the command line version. Implicitly, by doing so we also refer to the environment variable and the configuration file entry. 4.1.4
Important options
4.1.5Less important options
  • -subnet_a, -subnet_b, -subnet_c - These options correspond to Primary/Secondary/3rd IP Address Range in &tde4;'s preferences. They allow you to specify address ranges for the automatic network mode. By default, subnet_a refers to the entire local area network of your host. As mentioned, scanning the entire net can take a long time. For this reason it makes sense to restrict the set of ip-addresses to be scanned. Values for this option are given in the standard dot notation, e.g. 192.168.45.0/24, where /24 indicates that the first 3 numbers (24 bit) are fixed while the last one is iterated through. The &tde4mhpd; node will first scan address range a, then b and then c. Another example: Let us assume your local area network has the network address 172.16.0.0/12. Your camera department including &tde4; seats is located in the subnet 172.16.32.0/20 and you wish to add nodes in this subnet. Then, during startup the node will determine the default value for subnet_a as 172.16.0.0/12, but it makes sense to set it to 172.16.32.0/20 which is scanned within a few minutes.
  • -tmp_dir - The directory for temporary data. It should point to a directory on a local drive, not on an NFS-mounted volume. The default is /tmp, and we recommend to leave it as it is.
  • -logfile - The path to the logfile. By default this is located in /tmp. Alternatively, you could place it in /var/tmp, but wherever you choose it to be, it should be on the local drive of your host, not(!) on an NFS-mounted volume. Also, it must be a regular file, not a special one like /dev/tty or the like. The node will truncate the logfile automatically as soon as its size exceeds 1MB. The precise default value is /tmp/tde4mhp.u{uid}.log where {uid} stands for the id of the user running the node. We recommend to leave this as it is.
  • -loglevel - Logged messages have a level of "severity". For &tde4mhp; we adapated the official UNIX levels ranging from LOG_DEBUG up to LOG_EMERG, see man syslog for details. Default is LOG_NOTICE. Only messages with the given severity level or higher are logged, all others are discarded. In case of trouble, if you need support from us, we might ask you to set this to LOG_DEBUG so that we have a better chance to solve the problem.
  • -launched_by - &tde4; passes some string here to indicate that the node is controlled. If you start &tde4mhpd; as a standalone node, please leave this empty.
  • -fileserver_base - This is an option for future applications running on &tde4mhp; which need access to a file server.
  • -known_hosts - This is another hint for the automatic network mode. &tde4; uses this option and passes a comma-separated lists of hosts obtained from the license server FLCD. You can use this option for standalone nodes as well, but in general it will be easier for you to pass the FLCD host by option -flcd_host instead.
  • -host - When a node is started, it will determine its own ip-address by analyzing network devices and the routing table. If this should fail for what reason ever, you can bypass this mechanism by this option. In that case please specify the ip-address here.
  • -install_dir - As you have noted, some options specify paths in the file system, e.g. the directory for lens distortion plugins or the location of a configuration file. These paths can be absolute (i.e. starting with "/") or relative. If they are relative, they are interpreted as relative to the "installation directory" which can be specified by -install_dir. If you do not provide this option, the path to the executable &tde4mhpd; is used as base for relative paths. Our suggestion in order to keep it simple, is not to use this option and use absolute paths wherever paths are required.
4.1.5.1Configuration file The configuration filename is specified by passing option -config_file <path/to/configfile> or by defining the environment variable TDE4MHP_CONFIG_FILE. A configuration file has the following simple format:
<option>	<value>
<option>	<value>
...
<option>	<value>
Lines starting with '#' are ignored, i.e. comments are marked by leading '#'.
4.1.6
Summary
Environment variablesCommand line optionsConfiguration

file entry
Possible

values
TDE4MHP_HOST-hosthostip address e.g. 192.168.1.32
TDE4MHP_PORT-portportport number, default is 57431
TDE4MHP_CHIEF_HOST-chief_hostchief_hostip address
TDE4MHP_CHIEF_PORT-chief_portchief_portport number
TDE4MHP_PROCESS_MODE-process_modeprocess_mode"foreground", "background", "daemon"
TDE4MHP_NETWORK_MODE-network_modenetwork_mode"automatic", "explicit"
TDE4MHP_SUBNET_A-subnet_asubnet_anet address e.g. 192.168.1.0/24
TDE4MHP_SUBNET_B-subnet_bsubnet_bnet address e.g. 192.168.1.0/24
TDE4MHP_SUBNET_C-subnet_csubnet_cnet address e.g. 192.168.1.0/24
TDE4MHP_PLUGIN_DIR-plugin_dirplugin_dirpath/to/plugin/dir/
TDE4MHP_TMP_DIR-tmp_dirtmp_dir/tmp, /var/tmp
TDE4MHP_LOGFILE-logfilelogfile/some/local/file
TDE4MHP_LOGLEVEL-loglevelloglevelLOG_DEBUG, LOG_INFO, LOG_NOTICE, LOG_WARNING, LOG_ERR, LOG_CRIT, LOG_ALERT, LOG_EMERG
TDE4MHP_NUM_THREADS_COMM-num_threads_commnum_threads_comminteger number
TDE4MHP_NUM_THREADS_PRIV-num_threads_privnum_threads_privinteger number
TDE4MHP_LAUNCHED_BY-launched_bylaunched_bystring
TDE4MHP_FILESERVER_BASE-fileserver_basefileserver_base/mount/point/of/fileserver
TDE4MHP_HOME_DIR-home_dirhome_dirdirectory, like $HOME/.3dequalizer
TDE4MHP_KNOWN_HOSTS-known_hostsknown_hostscomma-separated list of hostnames
TDE4MHP_FLCD_HOST-flcd_hostflcd_hosthostname
TDE4MHP_FLCD_PORT-flcd_portflcd_portport number, default is 57423
TDE4MHP_CONFIG_FILE-config_file- n.a. -path
TDE4MHP_INSTALL_DIR-install_dir- n.a. -base for relative paths
- n.a. --help / help- n.a. -- n.a. -
LD_PLUGINS_3DE4- n.a. -- n.a. -Additional directory for lens distortion plugins
4.2
Requirements and recommendations 4.2.1The owner of tde4mhpd We recommend, not to run &tde4mhpd; as user root. No resources accessed by &tde4mhpd; require root privileges, and you are on the safe side if you run it as a standard user or even create a particular under-privileged user account. 4.2.2Environment variables $PATH and $HOME The node needs to know the full path of its executable file. This is determined by means of the environment variable $PATH, which must be present in the shell from which &tde4mhpd; is started. This is usually not a problem on UNIX-like systems in higher run levels (&linux; terminology), but if this is not the case, you can alternatively start &tde4mhpd; by its full, absolute path, so that the environment variable is not required.

&tde4mhpd; reads the environment variable $HOME and creates a directory .3dequalizer for non-temporary data. If for what reason ever, this environment variable does not exist, please use the option -home_dir and set it to /some/existing/directory/.3dequalizer. If $HOME exists, please do not use this option.

4.2.3
Shell commands A certain set of shell commands must be present, since they are invoked by &tde4mhpd;. These commands are
  • awk (POSIX.1)
  • grep (POSIX.1)
  • killall
  • kill (POSIX.1)
  • ps (POSIX.1)
  • uname (POSIX.1)
  • wc (POSIX.1)
  • /sbin/ifconfig (required on &osx;)
  • /sbin/ip (required on &linux;)
  • /sbin/route
Not all of these commands seem to be mandatory by the POSIX standard, however, according to our experience they are present on &linux; and/or &osx;. An easy way to check if all commands are available, is to use the which-command. Under &linux;:
which awk grep killall kill ps uname wc ifconfig ip route
and under &osx; (without ip):
which awk grep killall kill ps uname wc ifconfig route
4.2.4
Default route It is strongly recommended that each host running a node has a default route. Usually this is the case, nonetheless we mention it here for reasons of completeness. The default route is required because the nodes are identified by the ip-address of the interface associated to the default route. Example: /sbin/route generates the following output:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.211.55.0     *               255.255.255.0   U     0      0        0 eth0
link-local      *               255.255.0.0     U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         10.211.55.1     0.0.0.0         UG    0      0        0 eth0
Then, from /sbin/ip route get 10.211.55.1 the node will find out that 10.211.55.7 is the ip-address of the interface leading to the standard gateway:
10.211.55.1 dev eth0  src 10.211.55.7 
    cache  mtu 1500 advmss 1460 hoplimit 64
so, the ID of the node will be 10.211.55.7.
4.2.5
TCP, UDP, Ports, Firewall Since communication can go in both directions between all nodes of the &mhp;-network (where the roles of server and client may change), all hosts must allow incoming and outgoing TCP and UDP connections for the port used in the &mhp;-network. Please make sure, the firewalls on all participating hosts allow this. On &linux; methods for modifying the firewall may vary between distributions. In this section we will at least try to cover a few methods. Note that All methods in the following subsections require administrative permissions. Modifying the firewall will affect the security of the hosts in your network, so make sure you fully understand the consequences of these commands. 4.2.5.1¢os; On ¢os; the firewall can be modified using the command firewall-cmd. The option --permanent ensures that the reconfiguration will survive a reboot of the host. The option --zone could look different in your case, please consult the documentation in case of trouble.
linux> sudo firewall-cmd --zone=public --add-port=57431/tcp --permanent
linux> sudo firewall-cmd --zone=public --add-port=57431/udp --permanent
linux> sudo firewall-cmd --reload
4.2.5.2
&ubuntu; On &ubuntu; and its derivates there should be a command ufw which comes in quite handy. Assuming you are authorized to use sudo you could try the following: 1. Check status of the firewall
linux> sudo ufw status
If the firewall is inactive (which is not recommended) all ports are open and mhp should work. If the firewall is active, then: 2. Open tcp/udp-port 57431 (or which port ever you are using for your mhp-network)
linux> sudo ufw allow 57431
4.3
Limitations It is not clear how many nodes an &mhp;-network can have. We think there is an upper boundary of 256 hosts participating in the network for technical reasons, which we are however not able to verify or falsify. Also, there are practical reasons which advise, that the number of hosts should be lower. For example, instead of building one large network of 256 machines we suggest to build two networks made of 128 hosts or four networks made of 64 hosts, each network running on a different port in order to avoid interference. Whatever you try, it makes sense to start with a small number of machines and increase it step by step. Your feedback is appreciated... 4.4The html-interface Each node responds to browser requests on the same port they use for communicating with each other. Some of the html-pages describe the node they were requested from, others will give you information about the &mhp;-network. 4.4.1The page hosts.html This page gives you information about the entire network.
ColumnContent
host:port In this column you see the ip-address and the port number of each node. It is advisable to use the same port number for all hosts in the network. In &tde4mhp; hosts are not identified by their hostname, instead we use the ip-address of the interface pointing to the standard gateway (the default route). This makes the network resistent against spontaneous renaming by the DHCP server or the like.
hostname This column shows the human-readable hostname generated by the host (man 3 gethostbyname).
proc The column "proc" displays how the node was started, as foreground process, as background process or as a daemon. Running a node in process mode "foreground" is helpful for learning and debugging puposes. For daily use, console output for debugging is no longer required, and nodes should be run as daemons.
msg This field contains the last status report from network connection between chief node and all other nodes. Whenever hosts.html is requested, the chief will fetch properties from all nodes (including itself). After some timeout, all collected data are displayed. Failed connections lead to an error message here.
launched_as This column shows if this is a controlled node or a standalone node.
role This column tells you if the node is chief, or loyal (normal node). The third possible state is renegade which appears, when a node has left the network unexpectedly without checking out. Renegades should be removed or reintegrated into the network. When the network connection fails due to a timeout, the role is labelled unclear.
threads This column displays the number of threads dedicated to the community and the number of private threads. For standalone nodes it makes sense to make all resources available to the community. Since &tde4; is not running on these hosts, private threads are not needed.
uptime This column displays the time the node has been running.
version In the introduction we have already mentioned that only nodes having the same major version and subversion number can collaborate, so it's important to display the version here.
perf After each parameter adjustment, the client collects the times for processing the jobs are averaged and mapped into a relative performance values. These values are sent to the chief and averaged over the previous ten projects. When the performance values are displayed, the fastest host is labelled with "100%" and all other hosts accordingly.
op The operations column contains buttons for restarting or terminating a node. Loyal nodes have a button which turns the node into the chief. In mixed networks with controlled and standalone nodes, it makes sense to associate the chief role to a standalone node, in order to release the controlled node from network traffic.
4.4.2
The page properties.html This page is requested for a specific host you wish to receive information about. It also provides functions to modify the structure of the network.
ButtonEffect
RESTART Restart the node. The node will keep its settings. If it has been the chief before restart, it will be chief after restart as well, so this opreration should keep the network consistent. Technically, this is done by sending the signal SIGUSR1 to the node process. The operation makes sense, when the software has been updated. There seem to be problems sometimes on &osx;, when the executable is located on an NFS-mounted volume.
TERM This will terminate the node regularly. When the node has been chief before termination, the chief role is moved to another node and all other nodes are relinked to the new chief. Computational child processes are not terminated, so ongoing computations will not be affected. These processes can appear as "undefined" in the process list, but should vanish when computations are done.
RELINK... Your browser will ask you for an ip address and a port number. The node is then relinked from its current chief to the new chief given by the host data you have entered. If the node has been chief before the operation it will take all loyals over to the new chief, so this operation can be used to merge two smaller networks into a bigger one.
UNLINK... The node is unlinked from its current chief and forms a network on its own (a "monadic" network). If the node is chief, the network will not be modified. This operation makes sense if you want to exclude the node from network computations without terminating it.
SCAN NW... When the node is started in automatic network mode, a subprocess is started, which scans the local area network for other nodes. When this process is over (successfully or not) the scanner process terminates. By this button you can initiate the scanner process manually.
COLLECT... When the chief is terminated unexpectedly, the network is left in an inconsistent state; the remaining nodes are checked in at a chief node that ceased to exist, yet they are not smart enough to re-organize themselves. Relinking each single node by hand would be a lot of work. In order to speed this up, you go to the properties page of the chief to be designated and press this button. The node will try to collect nodes in the inconsistent network and form a new, well-defined network. So, this is kind of a "repair" button, but it should only be pressed, if there is need to repair something.
Figure 2: RELINK Figure 3: UNLINK Figure 4: COLLECT
4.4.3
The pages processes.html and processes_network.html A node creates other processes for performing various tasks. You can examine these processes in the browser interface, and terminate them in case of trouble. Processes can be examined for the entire network (as in the figure below) or for a single node. In the network view, responder processes are omitted. Figure 5: Processes running on hosts in the network In the following table all process categories are listed.
Process nameFunction
parentThe &parent; process waits for incoming tcp-requests. When a tcp-connection is established, it creates a thread or a process.
responderA &responder; process is created when an xml-, html- or any other http-related file is requested. Usually, processes of this type only exist for a less than a second.
computationalEach user doing computations on the node causes the creation of a &computational; process, i.e. there can be more than one &computational; at a time. Processes of this type create threads and use a lot of cpu-time. When no calculations are done in the &mhp;-network, there should not be any computational process. When the &parent; process is killed, &computational; processes will keep working, until their jobs are done.
newsThe chief host spawns a &news; process or each instance of &tde4; connecting to the &mhp;-network. The &news; process sends information about the current status of the network to &tde4;, which are then displayed in &tde4;'s GUI.
updater_udpThis process is spawned by the chief. It creates an udp-socket and waits for messages from all nodes. The &updater_udp; process will run as long as the chief node is running (and as long as the node is chief). There can be only one &updater_udp; process per chief. When (temporarily) there are more processes of this type the additional ones should terminate after a few seconds.
scannerIn network-mode automatic this process scans the tcp-network for a chief host and checks in at the node running there.
preforkDuring startup, i.e. before accepting network connections, the process is called prefork. Processes with this label will generally not appear in process lists, but you may encounter them in the logfiles.
undefined When a node is restarted, its subprocesses (e.g. computationals) keep on running. These subprocesses are unknown to the new node process and appear as undefined in process lists. They are no problem as long as they disappear after some time, say, when they are no longer needed for computations.
4.4.4
The page load.html This page lists all nodes participating in the &mhp;-network. In each row you see the community threads and the private threads of each node. This snapshot was taken, when six users where doing computations in the network. &tde4mhp; generates a colour for each user in order to visualize how threads are associated to the users (that's why it looks like Teletubbyland meets My Little Pony). You see that community threads are generally divided among some or all users, while the private threads are reserved for one single user working on &tde4;. The page is updated periodically after two seconds. Please note, that your browser will receive live data from the chief of the network. Each browser displaying this page will increase network traffic at the chief node (slightly). Figure 6: MHP at work 4.5Lens distortion plugins If you are using your own lens distortion plugins, &tde4mhpd; needs to know where they are located. You can pass the directory as option -plugin_dir or as environment variable TDE4MHP_PLUGIN_DIR or as entry in the configuration file. Additionally, for reasons of compatibility to &tde4; the directories given by the environment variable LD_PLUGINS_3DE4 are scanned for plugins, where LD_PLUGINS_3DE4 contains one or more directories, separated by ":". After starting &tde4mhpd; it's a good idea to check in the HTML interface, if all plugins are loaded correctly (http://myhost:myport/ld_models.html). The version number in column "header" refers to the version number in the header tde4_ld_plugin.h containing the base class tde4_ld_plugin, from which you have derived your plugin class.
5Miscellaneous 5.1Todo A command line based administration tool would be helpful. 5.2An incomplete list of useful shell commands UNIX-like operation systems provide a set of shell commands which can be helpful in build and maintaining networking applications like &tde4mhp;. Some of them might be located in /sbin on your machine, or require administrator rights. If you are missing your preferred tool here, don't hesitate to write us, we will add it to the list.
NamePurpose
curloften available on &osx;. non-interactive download of files (http,https,ftp).
ifconfigdoes a lot, useful for examining network interfaces
ipcsreport interprocess communication facilities status. The shared memory acquired by &tde4mhpd; will appear here.
killsend a signal to a process. If you need to kill a &tde4mhpd; process, please keep in mind that it's better to try -TERM (the default) instead of -KILL (-9). Only use -KILL if necessary, because &tde4mhpd; won't be able to clean up.
killallkill (more precise: send signal to) processes by name
lscpulist cpu information
lsoflist open files. This one produces a lot of output. Some options are particularly interesting, like e.g. -i which lists processes with open sockets. Or you may try lsof | grep tde4mhpd, in order to see what &tde4mhpd; is doing on your machine. Lens distortion plugins should appear in the output.
netstatthe Swiss knife in networking
nmap(obsolete) a port scanner. Should be available as a package in most &linux; distributions.
pingcheck if host is up and reachable
psreport process status
topdisplay processes
wgetoften available on &linux;. non-interactive download of files (http,https,ftp).
5.3Multi-host administration There is a number of administration tools for multiple hosts available. We are currently using &clusterssh; which comes in quite handy.
Document created with CDML from /server/devel/sdv/privat/uwe/source/tde4_doc/tde4_mhp/cdml/tde4_mhp.xml


© 2014 by Science-D-Visions.