201706, Grimmer, ASIAA
After 201611, Code Contributor:
Grimmer, Mark, Erh-chung (left), Chia-Jung, Kuo-Song, Adrianna, Jamie, Anthony
Tester & support scientist, product manager:
Kuo-Song
Get Feedback and help, thank you:
Susan, Jeff, Ville, Darrell, Anand, Erik, Pavol, David, Dirk, Chin-Fei, Chi-Hung, Andrew and user questionnaire participants.
The week agenda https://goo.gl/HTkXLd
Grimmer slide:
another doc: summary questions about casa/casaviewer
KuoSong :
New Features
Added items ?
Software flow:
Tried to use Agile/Scrum SW development and executed code review.
Code & Build:
Tests, bug fixes and improvement
Docs:
GitHub + ZenHub
Text
Lightweight issue function:
feedback, discuss
Task (feature, bug) management
+ agile
1. Tried to use the same/near version of 3-party libs on different platforms. Good for developing and testing. * (gcc/clang, gfortran compiler not fixed yet)
2. Three dependency
3. CARTA -> GPLv3+
4. packaging script are on https://github.com/cartavis/deploytask
5. building casa no needs another qwt, qt4, dbus, java, rpfits, after modifying CMakeLists.txt.
6. on Mac, all use homebrew, and backup some homebrew bottles https://github.com/cartavis/homebrew-tap
7. every plugin is working. On ubuntu, regionDS9 needs flex built by ourself
2.5.35 + 2.3 no. Apple default, build fail.
2.5.35 + 3.0.4 no Apple default flex + bison 3.0.4 (brew)
2.5.35 + 2.4.1 (yum install on centos 6 and works)
2.5.37 + 2.7-4 ok centos 7
2.5.37 + 3.0.4 ok macports (2.3.4)
2.6.0-11 +3.0.4 crash, mark ubuntu 16.04
2.5.37 (built) + 3.0.4, ok mark ubuntu 16.04
2.6.3 + 3.0.4 no homebrew , Mac
2.6.4 + 3.0.4 no new homebrew , Mac
2.6.1 + 3.0.4 latest macports, build ok but function not work.
2.5.37 (built) + 3.0.4 (homebrew) ok.
- Done: no need to setup config.json, embedded it, can setup if you want to customize
- no default image for desktop version.
- no two folders (~/CARTA, ~/.cartatavis/ -> uses ~/.carta/)
- change "dbPath": "/tmp/pcache.sqlite" to "dbPath": "~/.carta/pcache.sqlite"
- no starting script
1. ulimit -n 2000 -> use release build of qooxdoo
2. the path of log file and html can setup in code/config.json
3. use random port or open port in UI for python interface.
4. casarc & setupcartavis.sh may be can be setup in QProcess?
5. the script will launch CARTA desktop twice, need improvements.
git clone https://open-bitbucket.nrao.edu/scm/casa/casa.git trunk
cd trunk
git checkout 77a3c0170c895142883dc1b69c4996f430c9e8ec ## = 5.0.0-mas-193, 20170506
git submodule update --init casacore
cd casacore
mkdir build && cd build
## cmake part of casacore, on Mac, use system-python
make -j2
make install
cd ../../code
curl -O https://raw.githubusercontent.com/grimmer0125/tmp/master/casacodereduce3.diff
git apply casacodereduce3.diff
mkdir build && cd build
## cmake part of casacode, now only build libcomponents, libimageanalysis, libstdcasa
make -j2
# in CasaImageLoader.pro
unix:!macx {
QMAKE_RPATHDIR+=$${WCSLIBDIR}/lib
}
else {
## will improve mac later
}
# on Ubuntu
# wget ftp://alma-dl.mtk.nao.ac.jp/sakura/releases/latest_src/libsakura-4.0.2065.tar.gz
## Use our own modified version,
## since the original source code will not be compiled OK by gcc 5.4, gcc 4.8 seem neither
## the below snippet is in buildcasa.sh
git clone https://github.com/grimmer0125/libsakura
curl -o gtest-1.7.0.zip -L https://github.com/google/googletest/archive/release-1.7.0.zip
unzip gtest-1.7.0.zip -d libsakura > /dev/null
cd libsakura
ln -s googletest-release-1.7.0 gtest
mkdir build
cd build
cmake ..
make
make apidoc
sudo make install
# on Mac
# the below is in https://github.com/CARTAvis/carta/wiki/Install-Third-Party-For-CARTA-CASA-On-Mac
#For building libsakura
brew install doxygen
brew install eigen@3.2 ## eigen is 3.3.3 but its 3.3.3 is not compatible with libsakura 4.0.2065
brew install log4cxx
git clone https://github.com/grimmer0125/libsakura
wget -O gtest-1.7.0.zip https://github.com/google/googletest/archive/release-1.7.0.zip
unzip gtest-1.7.0.zip -d libsakura
cd libsakura
ln -s googletest-release-1.7.0 gtest
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=BOOL:OFF -DSIMD_ARCH=SSE4 \
-DEIGEN3_INCLUDE_DIR=/usr/local/Cellar/eigen@3.2/3.2.10/include/eigen3 ..
make
make apidoc
sudo make install
cd ../../
CircleCI (Docker) | Travis-CI (Mac build) | |
---|---|---|
Pro. | 1. Faster 2. ssh login into build container 3. 3/4GB build artifacts 4. slack integration |
1. Free build minutes for open source project 2. Free Mac build 3. slack integration 4. Homebrew |
Con. | 1. Limited free build time, 1500min /1 month 2. Mac build needs paid plan |
50 mins limitation for each build stage |
Google Drive
Node.js app
if branch name is *release*, ci_mac_buildcarta.sh:
if [ "$TRAVIS_BRANCH" != "" ] && [[ $TRAVIS_BRANCH == *"release"* ]]; then
echo "carta build is release type"
qmake -config release NOSERVER=1 CARTA_BUILD_TYPE=release $cartawork/CARTAvis/carta -r
else
echo "carta build is dev type"
qmake NOSERVER=1 CARTA_BUILD_TYPE=dev $cartawork/CARTAvis/carta -r
fi
Fix the QtWebKit.framework & QtWebKitWidgets.framework structure
Modified QtWebKit.framework & QtWebKitWidgets.framework so that there is "Versions" directory
with numbered version "5" and a "Current" link to that. Symlinked "Resources", "QtWebKit",
and "Headers" to the "Versions/Current" directories.
configuration
step-1: install and launch pureweb on ubuntu
step2: use
for CARTA, one important package: sudo apt-get -y install Xvfb
samples in SDK: http://docs.pureweb.io/sdk4.3/content/explore/build_the_samples.html#
step3: place built-Qt binary & libs(.so) and JS client code to appropriate places. Then setup some config file(s)
lrwxrwxrwx 1 developer developer 42 Dec 23 14:22 buildDir -> /home/developer/src/CARTAvis/buildindocker
lrwxrwxrwx 1 developer developer 51 Dec 23 14:22 html5 -> /home/developer/src/CARTAvis/carta/VFS/PureWebDevel
lrwxrwxrwx 1 developer developer 20 Oct 3 2014 plugins -> buildDir/cpp/plugins
lrwxrwxrwx 1 developer developer 26 Oct 3 2014 server -> buildDir/cpp/server/server
developer@23404304159d:~/src/PureWeb/4.1.1/Server/webapp/Apps/Skeleton3.release$
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<bean id="licenseManager" class="pureweb.process.DefaultLicenseManager"/>
<bean class="pureweb.process.SocketProcessFactory">
<property name="useFullDuplex" value="true"/>
<property name="connectionBarrier" ref="applicationConnectionBarrier"/>
<property name="applicationRegistry" ref="applicationRegistry" />
<property name="licenseManager" ref="licenseManager"/>
<property name="application" value="Skeleton3"/>
<property name="description" value="The server side version of the skeleton3."/>
<property name="directory" value="${catalina.home}/../webapp/Apps/Skeleton3"/>
<property name="executable" value="${catalina.home}/../webapp/Apps/Skeleton3/server"/>
<property name="arguments">
<list>
<!-- <value>/PureWeb</value> -->
</list>
</property>
<!-- attempt to append to LD_LIBRARY_PATH failed :( (pavol)
<property name="environment">
<map>
<entry key="LD_LIBRARY_PATH" value="${systemEnvironment['LD_LIBRARY_PATH']}:/home/calsci/Qt53/5.3/gcc_64/lib"/>
</map>
</property>
-->
</bean>
<bean class="pureweb.cluster.SupportedClients" scope="prototype" id="supportedClients">
<property name="supportedClients" ref="supportedClientsMap"/>
<property name="application" value="Skeleton3"/>
<property name="clientMap">
<map>
<entry key="ios" value=""/>
<entry key="android" value=""/>
<entry key="html5" value="Apps/Skeleton3/html5/server/serverIndex.html"/>
</map>
</property>
</bean>
<!-- Adds the link under the Apps section of the PureWeb Web Application -->
<bean class="pureweb.process.PluginLink">
<property name="registry" ref="pluginLinkRegistry"/>
<property name="supportedClients" ref="supportedClients"/>
<property name="name" value="Skeleton3"/>
<property name="description" value="Skeleton with qooxdoo."/>
<!-- Optional. If specified the link generated will be as below. Otherwise all client links will be generated dynamically. -->
<!-- <property name="path" value="/pureweb/view?name=ScribbleAppQt&client=silverlight"/> -->
<property name="image" value="/themes/pureweb/default.png"/>
<property name="role" value="ROLE_PUREWEB_USER"/>
</bean>
</beans>
Skeleton3-plugin.xml
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0">
<title>CARTA</title>
<!--<base href="/Apps/carta/html5/"/>-->
<script type="text/javascript">
function getURLParameter(name) {
return decodeURIComponent((new RegExp('[?|&]' + name + '=' + '([^&;]+?)(&|#|;|$)').exec(location.search)||[,""])[1].replace(/\+/g, '%20'))||null
}
var ref = "/Apps/" + getURLParameter("name") + "/html5/";
// var ref = "/Apps/" + "Skeleton3" + "/html5/";
console.log( "dynamically setting base href to", ref);
document.write("<base href='" + ref + "' />");
</script>
<script src="server/pureweb.min.js"></script>
<script src="common/libs.js"></script>
<script src="common/CallbackList.js"></script>
<script src="server/serverConnector.js"></script>
<script src="common/skel/source/script/skel.js"></script>
<!--<script src="common/skel/build/script/skel.js"></script>-->
</head>
<script>
function getURLParameter(name) {
return decodeURIComponent((new RegExp('[?|&]' + name + '=' + '([^&;]+?)(&|#|;|$)').exec(location.search)||[,""])[1].replace(/\+/g, '%20'))||null
}
console.log( "location=", location);
console.log( "location.search=", location.search);
console.log( "name=", getURLParameter("name"));
</script>
<body>
</body>
</html>
71 closed pull requests
- fix python interface (was broken > 6 months) #2
build
- plugins, qt 5.5, gcc 5.4, new namespace of casa libs, docker , circleci, travis-ci
misc.
App name: "CartaVis" --> "CARTA"
Mac High-DPI
Add "clipping, region" in toolbar
better font
add help, coypright and license, version, does not show address bar
built-in config.json #57
background color
Window - centered, larger
layout
- Add missing check status of layouts.
- Add different predefined layouts: Default, LineAnalysis (profiler), Histogram, Image (ImageZoom and ImageContext), Custom
fix reading casa image file #19
fix reading mom file #35
add icon for region, image files (miriad, casa image file, fits) #35 #41
update its content #167
- show beam size with dynamical precision, #25
- rename "Right Ascension Range" --> "RA Range", "Declination Range" --> "Dec Range", "CasaImageLoader" --> "ImageViewer" #60
clip
rename "Recompute Clips on New Frame" -> "Per frame"
default 100% clipping
- Show default unit, before it always was Jy/beam. #55
- fix reverse function
- add 3rd and 4th power.
- a filter to reduce the number of colormaps from matlplotlib #18
- fix drop frame #15
- add fileList in animator #81
animator & colormap - default use stoke-I to calculate & add UI to switch stokes #115
imageViewer
Show the whole image in the imageLoader window when open a new image file. #52
Fix the bug that the image is not centered. (casa) #59
auto select image loader to show menubar (UI) ! #141
No more black image shown about 0.x second when launching CARTA. #141
if z-axis is not channel, show image correctly #12
- add currentFileName, quantile-info (#162)
- more precision coordinate #23, #31, #35
- cursor with radio velocity #43
grid line
- sharpen grid line of desktop ver. (ast)
- default no grid line
- no more default J2000 #97
- use casacore to parse files then feed data to ast lib to show correct grid line for non J2000 system. Also try to show rotated images #40
image setting
- No default enabled "Canvas -> All images". (also "Apply to all images": permutation and coordinate system of a image file`)
- correct Label/ImageSetting (offset, velocity (km/s)) for PVImage #42
- the format in Image Setting -> Labels will change according to the current X and Y axis. #50
- pvimge shows offset, velocity in image setting (use AxisMapper class), #144
- No more imageSetting shown about 0.x second when launching CARTA #141
grid line+ imageSetting:
set proper default coordinate system (use AxisMapper) #163
grid line+ imageSetting + cursor:
Add a basic spectral system in carta #164 (not merged yet)
region
- Cosmetic (Jamie) #149
crash
- A file -> B file, incompatible axes, crash #38
- use cmd + shift +r to refresh CARTA desktop without re-starting it. #146
- {nan, nan} crash in quantile #12
performance
- 2d old in memory cache (not leveldb) hit or max/min of 3d cube in the same plane #49
- reuse raster image #9
- grid line - ignore keywoards #16, #40
- reuse grid line
- New Cache (Pavol, Adrianna): percentile and quantile #8
- fixed memory leaking when image is permuted #10
After
We have fixed a bug so that you can setup JavaScript breakpoints in UI initialization stage.
Steps:
1. Launch CARTA desktop
2. open JS console
3. insert breakpoints or debugger;
4. shift + r to reload UI (CARTA is not re-launched)
5. it will stop at that point.
80 issues = 14 (milestone1) +
27 (milestone2, Sep, release 0.9, basic functions) +
save: introduce later
histogram: not shown completely OR not shown correctly
profiler: should not see a "spectrum" for single channel image
incorrect unit for y-axis
..
..
unlisted issues (e.g. sometimes crash but some of them not reproduceable)
python interface - design & incomplete command set
Architecture issues
UI/UX
spec, test & design (user research & competitor analysis etc)
Performance
Build
In general software company (e.g. microsoft or startup), coding vs testing+bug fix is 1:1 or 1:2. Also CARTA is a UI program, more need tests. Wireframe, UI spec, Soft. document, testing flow/cases/documents are needed.
if users change move B file as A file name, it will use A's original cache as B's cache, wrong.
Suggested solution: Is it possible to add file date as a part of a unique identifier/key?
1. fixed drop frame issue (remove some of timer in CARTA)
2. the current design does not keep same fps and possibly make duplicate rendering -> propose to use a rendering timer + playing buffer + timestamp on playing client to play animators, just like a normal video player design.
3. animator will draw background image file. <-discuss
DrawStackSynchronizer::_getLoadableData
side effect: the grid line is wrong when ra vs channel
// carta/cpp/plugins/WcsPlotter/AstWcsGridRenderService.cpp
for ( int i = 0; i < infoCount; i++ ){
if ( displayInfos[i] != m_axisDisplayInfos[i] ){
m_axisDisplayInfos[i] = displayInfos[i];
- m_vgValid = false;
+ //m_vgValid = false;
}
}
screenshot in #58:
catch( const std::exception & ex ) {
qCritical() << "Exception(std::runtime_error)!" << ex.what();
}
1. rotation issue of pc matrix, @Dirk
2. Carta can not show the right axis information of "green.fits". Erh-chung:
I noticed that green.fits is considered as linear coordinates because of bugs in the latest version of wcslib[v5-1x]. If we build casacore with wcs-4.25, it will show correct coordinate system.
3. [region] crash in LCBox.cc https://github.com/CARTAvis/carta/issues/161
save: is not WYSIWYG & have bugs (e.g. no region, clip, zoom level), no pdf file export
Proposal:
1. On JavaScript client side, no above save options, just save current imageViewer window to jpg.
2. Send jpg to cpp/server side
3. convert jpg to any file formats
4. Send back to JavaScript/User pc side.
Some are related to arch.
process.response.timeout=1300 #13
process.shutdown.timeout=1300 #13
client.activity.timeout=1300 #13
cluster.status.timeout=1300 #13
cluster.response.timeout=1110 #11
1. table of js, python command
2. python:114. js:178. (overlap:60)
3. no interactive way, dislikecasaviewer, more like coding.
4. [disadvantange] it is connecting toDekstop version, not server version.
a.pureweb supply oauth, basic auth.
b. or replacepureweb, then use these auth module on our server.
5. python interface can not be notified changes from CARTA if users do some action on UI
6. not straightforward to add commands, need to addcpp code. see below arch issue page
7. example to add a command: https://goo.gl/azZ911, most of the work is in ScriptFacade.cpp.
Disadvantange:
just clients <-WebSocket-> Local/Remote Rendering Server written by our own. And UI related (default) values are handled by UI (JavaScript), cpp is for algorithm. Give up shared session of PureWeb (too much side effect). Simplifying is always better.
To maintain desktop and server is a big overhead. Developers need to do developer-test them before someone to test/demo. Also, it is not easy to test Server version for Mac's developers, although we have docker. Suggest to only support browser <-> Server version.
If keep desktop but no browser, cpp side is still possible to exist in local/remote Server, not in desktop app. So that the architecture still becomes simplified.
Candaite libs:
The ideal situation is to get UI/UX designers which having science sense, also get UI engineers.
Grimmer's questions:
9. what is this?
/**
* Returns whether or not this window supports establishing a two-way
* link with the given plug-in.
* @param pluginId {String} the name of a plug-in.
*/
isTwoWay : function(pluginId) {
10. setup a minimized level of zoom out , which is just to fill the imageViewer.
11. what is the design purpose of shadow effect of "grid line"?
12. https://github.com/Astroua/CARTAvis/issues/185, @adrianna
13. https://github.com/Astroua/CARTAvis/pull/143, @adrianna
14. https://github.com/Astroua/CARTAvis/issues/96#issuecomment-152553866 @Susan