XmlUserManager is a small extension of the built in example SimpleUserManager. Instead of reading user definitions
only from proxy.xml it can fetch an external xml file from any url. This can be a simpler alternative to a full
database usermanager, when maintaining the same set of users in a cluster of multiple proxies.
- XmlUserManager uses the exact same xml format as SimpleUserManager (see proxy-reference.html).
- It will accept user definitions included in proxy.xml, just like SimpleUserManager. These are parsed before any
attempts are made to get external definitions, making it possible to have some local users in addition to the main
user db.
- If the same username exists both in proxy.xml and in the fetched user file, the local definition will be overwritten.
NOTE: Errors in the user file will not prevent the proxy from starting, it will only log warnings if unable to
fetch/parse the file. Local users in proxy.xml will still be available.
Should the user file become temporarily unavailable or broken, the proxy will keep using the last known working one.
To see exactly what XmlUserManager is doing, use log-level FINE.
As of 0.8.8 it is possible to specify multiple source urls, using the following format:
http://192.168.0.5/users.xml
asdf22
https://admin:[email protected]/users.php
file:///tmp/users.xml
5
Cache-sharing with the csproxy is a powerful tool that can be used in a number of different ways. Before contemplating
the various possible scenarios, the basic cache model needs to be understood.
Whenever a proxy node receives a new ecm request from a client, it will query the cache to see if this is the first
time this particular request has been received. One of three things can happen:
1. The ecm has been received before, and a cw reply has already been fetched from a card. This causes an instant hit.
2. The ecm has been received before, but the card transaction is still ongoing and the reply not yet available.
This causes the request to be held in the cache for a maximum of 'max-cache-wait' seco$$$. As soon as a reply is
available all threads waiting for it will be released with their own copy of the cw.
3. The ecm has not been seen before, no card transaction is pending for it. This will place the ecm in a list of pending
requests, and it will be up to the proxy service mapper and connection manager to find a card to handle it.
The cache knows nothing about which ca-system the ecm belongs to, what sid it is for, or who is asking for it. It will
simply check its lists of already handled ecm -> cw pairs, and the list of pending ecm requests and see which (if any)
contains the new ecm. If two different profiles, or multiple services within one profile, happen to use the same ecm's
the cache will score hits. The chances of a false positive are insignificant enough to be safely ignored.
When using remote cache-sharing, all of the above remains pretty much the same. However, whenever something is added
to the list of pending requests or requests with cw replies, the cache will broadcast the new addition to the remote
proxy (or proxies) that it has been configured to talk to. This broadcast can be done in several ways, but all involve
udp communication and proprietary java object serialization.
Note that in any cache cluster, there can still be occasions when two or more proxies will attempt to query a card for
the same ecm. This will occur when the ecms arrive at both proxies at exactly the same time (which is of course often
the case). Depending on the roundtrip ping time between the proxies, this can be enough to reduce the effectiveness
of the cache significantly. Typically though, you have multiple proxies for the sake of redundancy, you probably want
the same ecm processed in several places so that if one proxy fails to produce a reply in time, another may succeed.
That said, it is possible to achieve a strict synchronization between cache-instances by using the arbitration feature,
which introduces a negotiation procedure for each ecm, to determine which proxy is best suited to handle it (and then
only that proxy will proceed with forwarding, the others will wait). See sync-period below.
NOTE: As of 0.8.13 the ClusteredCache no longer uses default java object serialization for the transport protocol.
The new custom protocol is briefly documented in ClusteredCache.java (it should be about 20-40 times more efficient).
------------------------------------------------------
The following are settings are available, see proxy.xml for separate examples:
- To use the ClusteredCache as the cache-handler for the proxy, use the class name: com.bowman.cardserv.ClusteredCache
Changing cache-handlers requires a proxy restart.
19
7
- Inherited from DefaultCache (see proxy-reference.html).
peer.proxy.host.com
54278
- One way of specifying where to send cache-updates, when there is only a single target proxy.
230.2.3.2
2
54278
- Another way to configure targets for cache-updates. Multicast typically only works in a LAN environment.
http://cstracker.host.com/list.enc
secretkey
10
- A third (and perhaps the best) option for configuring multiple targets for cache-updates. The ClusteredCache will
fetch a plain text list of hostnames and portnumbers, and send updates to every entry in the list. The list can be
automatically fetched at regular intervals (or if tracker-update is 0, only when proxy.xml is modified/touched).
This approach allows proxies to be added to a cluster without having to modify the configuration of already existing nodes.
The list must have the following format (# are comments):
# ClusteredCache list file. Syntax: hostname_or_ip:udp_port_nr
proxy1.host.com:54278
proxy2.host.com:54278
192.168.0.3:54275
The list can be stored anywhere, as long as it can be accessed via url (e.g file://, http, https, ftp).
Optionally, the list can also be blowfish encrypted using the included tool fishenc.jar (found in lib, java -jar fishenc.jar).
If encrypted, tracker-key must be correctly set.
54278
- The UDP port where this ClusteredCache instance will listen for incoming updates.
csproxy3.host.com
- The external hostname or IP of this ClusteredCache. This is only needed when using the above tracker setup. The name
should match the one in the tracker list, so the cache instance can identify itself in the list and avoid sending itself
updates.
true
- Set debug to true to enable additional cache information in the status-web. This can impact performance, use with care.
false
- If configured to send to one or more remote caches, this controls whether the names of the connectors are censored in
the outgoing cache updates (will appear as remote: unknown to the target). Only makes sense when dealing with untrusted
proxy peers.
0
- Set this > 0 to enable the strict arbitration procedure. For example if you set it to 200, the ClusteredCache
would use 200 ms for every new (previously unseen ecm) to wait and synchronize with as many other proxies in the cluster
as possible, and determine who is best suited to handle it. Only this proxy would proceed with a forward to a card.
This adds 200 ms to every single transaction, but should ensure that a cluster of proxies will only ask one card in one
proxy, once, for the same ecm. Probably only usable in a cluster where all nodes are fully trusted, and where the
network is reliable with fairly fixed ping times and no congestion.
------------------------------------------------------
NOTE: It is possible to set up the ClusteredCache without any remote targets (receive-only-mode). If no remote-host/port
is set using any of the various methods then the default behaviour will be to not attempt any sending of updates.
This is useful when creating a cache-only proxy node, receiving updates from multiple other proxies but sending to none.
It can also be used to augment local cards with additional services (which will only be available through the cache).
CSP Plugins
-----------
As of 0.8.6 the proxy has a plugin framework. This is another one-hour hack so don't read too much into it.
NOTE: With 0.8.10 it is also possible to add connector implementations using these same methods, although connectors
wouldn't be loaded under the plugins section (see README.$$$$$Connector.txt).
Some ideas for plugins:
- CA emulation, have the proxy read/fetch/use the static keys and the clients wont have to.
- Your own simplified CS protocol.
- BISS.
- Log aggregation, have the proxy receive udp/syslog events from servers and clients to aid troubleshooting.
- Fault management, triggering nagios or zabbix alarms on critical errors.
To create a new plugin, first read README.Compiling.txt and ensure you can successfully build the proxy itself.
The source code itself is the ultimate documentation and to find a place to start, you can use the following param when
starting java: -Dcom.bowman.cardserv.util.tracexmlcfg=true
That will keep track of where all proxy.xml accesses are made from in the code, and dump it to file when
you request it via the admin section of the status web.
Then use the following procedure:
1. Copy one of the existing plugin directory trees and rename it (e.g MyTestPlugin).
2. Edit build.xml in the plugin dir and search/replace the old name to match yours.
3. Place any extra dependencies your plugin will need in the lib dir. Remove any jars that are not needed.
4. Start editing the source, renaming the main plugin class and file to match your new name.
5. Run ant in the plugin dir to compile and build the jar (it e$$$ up in dist).
Place the jar in proxy-home/plugins, then add the config elements for the plugin to proxy.xml, e.g:
...
// plugin specific config here
...
The plugin lifecycle is as follows:
1. The main plugin class (that implements ProxyPlugin) is instantiated.
2. configUpdated() is called with the settings specified for this plugin in proxy.xml.
3. Assuming configUpdated() didn't throw any exceptions, start() is called, with a reference to the proxy istself.
4. Next time proxy.xml is touched or changed, stop() will be called allowing the plugin to cleanup before unload.
For most plugins it should be possible to replace the jar file and update/touch proxy.xml to have the new version loaded
without restarting.
NOTE: As of 0.8.11, plugin jars are watched for changes, and automatically reloaded when replaced.
The plugin api is fairly primitive, here's a quick guide for a single class example:
package com.bowman.cardserv;
// You can use any package, but there could be some protected methods only accessible from this one in the main classes.
import com.bowman.cardserv.interfaces.*;
import com.bowman.cardserv.util.*;
import com.bowman.cardserv.rmi.*;
import com.bowman.cardserv.web.*;
// Depending on what you intend to do, different parts of the proxy source needs to be imported.
import java.io.*;
import java.util.*;
public class MyTestPlugin implements ProxyPlugin {
// The main class of the plugin must implement this interface: com.bowman.cardserv.interfaces.ProxyPlugin
// If you want the plugin to have a say in connector selection, also implement: com.bowman.cardserv.interfaces.CwsSelector
// If you want the plugin to filter replies from connectors (dcw's) implement: com.bowman.cardserv.interfaces.ReplyFilter
// Methods outlined below...
public void configUpdated(ProxyXmlConfig xml) throws ConfigException {
// Whenever proxy.xml is changed the plugin will be discarded and reloaded. Settings from proxy.xml available here.
// The plugin should verify that they make sense and throw a ConfigException if they dont.
}
public void start(CardServProxy proxy) {
// Called after configUpdated(). Tells the plugin to initialize everything and start any background jobs etc.
// If it needs access to the proxy it should store the reference passed to this method.
// This reference can be used to get access to most of the functionality, see MessagingPlugin for one example.
// Any control or status comma$$$ should be registered here.
}
public void stop() {
// Called before unload is attempted (every time proxy.xml changes, or when the plugin jar is replaced).
// The plugin should stop all threads and remove any references to itself that it might have registered elsewhere.
// Any control or status comma$$$ should be unregistered here.
}
public String getName() {
return "MyTestPlugin"; // Displayname for the plugin
}
public String getDescription() {
return "A test plugin."; // Description
}
public CamdNetMessage doFilter(ProxySession proxySession, CamdNetMessage msg) {
// Called with every single message sent to the proxy from a client session, or from the proxy back to the sessions.
// The plugin can modify the message, return something else entirely, or block it (by doing msg.setFilteredBy("Reason text")).
// See the LoggingPlugin or EmmAnalyzerPlugin for examples.
return msg; // = do nothing
}
public byte[] getResource(String path, boolean admin) {
// This allows the plugin to export content to the httpd
// The following code ensure that anything placed in /web in the plugin jar file is available via the proxy web.
// Files are accessed using http://proxy.host.com/plugin/PluginName/...lename.ext
// The admin flag indicates if the user logged into the web as an admin user.
if(path.startsWith("/")) path = path.substring(1);
try {
DataInputStream dis = new DataInputStream(getClass().getResourceAsStream("/web/" + path));
byte[] buf = new byte[dis.available()];
dis.readFully(buf);
return buf;
} catch (IOException e) {
return null;
}
// Note that user login will be required to access anything this way, with one exception:
// The plugin will be queried for a file called "load.js" whenever anyone accesses the default status web (cs-status).
// This file allows the plugin to hook itself into the javascript in cs-status.war. See GeoipPlugin for an example.
}
public byte[] getResource(String path, byte[] inData, boolean admin) {
// Same as the above method, except for http POST instead of GET.
// Can be used to allow file uploads from the web to a plugin (or any custom data from the client side scripting).
return null;
}
// If your plugin implements the CwsSelector interface, the proxy will call this method for every incoming ecm request.
// The call includes the session where the message originated, and the connectors (set of name Strings) that the proxy
// thinks are valid choices to handle this request. Your plugin can remove names from this list based on the contents
// of the CamdNetMessage data, or based on the properties of the user associated with the session.
/*
public Set doSelection(ProxySession session, CamdNetMessage msg, Set connectors) {
return connectors;
}
*/
// If your plugin implements the ReplyFilter interface, this method will get called for every connector reply (dcw).
// This happens before the proxy does anything with the reply, and allows the plugin to look for and remove bad dcw's.
// To change a suspicious dcw reply into a cannot decode, do msg.setCustomData(new byte[0]).
// To silently block a reply entirely (probably a bad idea) return null instead of the msg.
/*
public CamdNetMessage doReplyFilter(CwsConnector connector, CamdNetMessage msg) {
return msg; // = do nothing
Compiling CSP
-------------
The proxy is pure java and can be compiled on any platform that has a java sdk available.
You'll need the following:
- The Java SE sdk and compiler (j2sdk) 1.6 or later: http://developers.sun.com/downloads/new.jsp
NOTE: As of 0.8.10 java 1.6 is required to compile everything, but 1.4 is still enough to run the proxy.
Sun only provides downloads for linux/win/solaris, osx users look to apple: http://developer.apple.com/java/download/
- Apache ant (xml makefile system) 1.6 or later: http://ant.apache.org/bindownload.cgi
- If you're on windows, you should consider using cygwin to get a proper bash shell: http://www.cygwin.com
- If you plan to change or fix something, a basic understanding of java concepts will help:
http://java.sun.com/docs/books/tutorial/
The proxy comes with project files for Intellij IDEA 7, but you don't need this to compile (its a commercial product):
http://www.jetbrains.com/idea/index.html
NOTE: There is now a open source community version as well, which works perfectly with the csp project files:
http://www.jetbrains.com/idea/nextversio...a_ide.html
Ant may require you to set the environment variable JAVA_HOME so that it points to your j2sdk install dir.
Once j2sdk + ant is installed (and their respective /bin directories added to PATH) you can build the proxy using the
following procedure:
$ tar xvzf cardservproxy-0.8.13-src.tar.gz
$ cd cardservproxy-src
$ ant
This runs the default build target "tar-app" in the build.xml file. This target compiles all classes, creates
the jar/war files and finally tars the installation directory structure.
Each output file e$$$ up in ./dist
NOTE: tar-app will attempt to build the example plugins (which may have dependencies). If you don't want these then
run ant with: -Dskip-plugins=true
--------------------------------------------------------------------------------------------------------------------
The source distribution is arranged as follows:
build - Temp directory for the installation distribution (dir structure for cardservproxy.tar.gz).
classes - Temp directory for compiled .class files.
config - Example configs and reference.
dist - Temp directory for generated distribution files.
lib - Dependency jars needed to compile (but not to run, you'll need to copy the jars from dist to lib to do that).
NOTE: Source for these dependencies is available on request (go find bowman on efnet or bow on freenode).
etc - Misc resources.
src - Java source files and other resources that will end up in cardservproxy.jar.
trtest - Files needed for the executable fishenc.jar (manifest only, all classes are in cardservproxy.jar).
web - Files for the status web monitoring gui (see README.HttpXmlApi.txt for details). E$$$ up in cs-status.war.
plugins - project dirs for the example plugins (each one laid out roughly like the main dir for the proxy source).
jsw-win32.zip - Template for a java-service-wrapper windows setup (running java and the proxy as windows service).
CardServProxy.ipr CardServProxy.iml - Intellij IDEA project and module files.
build.xml - Apache ant makefile.
NOTE: While the java source may seem somewhat organized at first glance, it's really a mess. It's the result of years
of organic growth (with contributions from many people) with little or no testing or oversight. Don't assume anything
in there is well thought through, if something seems broken - it probably is.
Available status comma$$$ (as of 0.9.0):
----------------------------------------
- proxy-status: general usage stats
optional parameters: profile (show info for this profile only)*
- ca-profiles: list of all defined profiles, and their listen ports
optional parameters: name (name of the profile to retrieve, omit for all)*
- cws-connectors: list of all defined connectors and current usage stats for each
optional parameters: name (name of the connector to retrieve, omit for all)*
profile (list only connectors within specified profile)
- proxy-users: list of all active user sessions, and their current status
optional parameters: name (user name to retrieve, omit for all)**
profile (list only sessions within specified profile)
- cache-status: usage stats from the current cache implementation
optional parameters: none
- error-log: the last 40 (by default) events of general interest
optional parameters: profile (list only events related to specified profile)
- user-log: the last 100 transactions results for a specific user (requires debug="true" for the profile)
optional parameters: name (list transactions for named user instead of calling user, admin only)
- user-warning-log: the last 40 transactions with attached warnings (aggregated if the same warning occurs often)**
optional parameters: profile (list only events related to the specified profile)*
- all-services: all services found on all connectors (requires service mapping and a services file)
optional parameters: profile (list only services for specified profile)
- watched-services: currently watched services, based on active user sessions (requires a services file)
optional parameters: profile (list only services for specified profile)
- export-services: dump of the current internal state for the service mappers (which sids can and cannot decode).
optional parameters: name (export map only for the specified connector, omit for all)
- fetch-cfg: returns a copy of the current proxy.xml (NOTE: response is without the cws-status-resp wrapper element!)
optional parameters: none
- ctrl-comma$$$: lists available control comma$$$, and their options
optional parameters: none
- status-comma$$$: list available status comma$$$, and their options
optional parameters: none
- last-seen: lists information about disconnected or removed users (seen before by the session manager)
optional parameters: name (only show seen info for the specified user, omit to show all)**
* By default, only profiles (or connectors within profiles) that the calling user has access to will be shown.
** All comma$$$ are available to any user, however only admin users will be able to retrieve information about
other users. A non-admin user will receive only their own data, regardless of the name parameter.
As of 0.8.6 status-comma$$$ are also dynamic, so any piece of code (plugin or otherwise) may add new ones or override
the function of a command in the above built-in set.
Available control comma$$$:
---------------------------
This list is dynamic (comma$$$ can be added by plugins and extensions).
Run the ctrl-comma$$$ status command as an admin user to obtain the current list, e.g:
http://proxy-host:8082/xmlHandler?command=ctrl-comma$$$
Built in comma$$$:
- reset-connector: clear the service map for one connector (delete all cannot-decode entries causing new probes)
required parameters: name (connector name)
- reset-service: clear the service map entry for one service.
required parameters: id (service id, by default in decimal format, use 0x prefix when specifying hex value)
profile (name of profile that contains this service)
- retry-connector: attempt re-connection for one connector (also temporarily enables disabled connectors)
required parameters: name (connector name)
- kick-user: close all sessions for the specified user
required parameters: name (user name)
- osd-message: send newcamd osd message to all sessions for specified user (or for all users), where clientid is mgcamd or acamd
required parameters: text (message text, avoid special characters)
optional paramaters: name (user name)
- clear-warnings: delete all user transaction warnings
- clear-events: delete all CWS events
- shutdown: stop this proxy node
Remote config updates:
----------------------
It is possible to deploy an updated proxy.xml via HTTP post, using the target-url: /cfgHandler
To fetch the current config xml, use the status command fetch-cfg (admin user required).
When posting an updated config to /cfgHandler, use the same approach as for Method 1 below, with two exception:
- The new config xml is to be posted as is, no cws-status-req or cws-command-req wrapper elements.
- As a consequence, only HTTP basic auth can be used for the login (no way to specify session id in the xml).
The response xml will be a single element:
(proxy -> client)
Command method 1 - HTTP post
--------------------
The default target url for both status and control comma$$$ is: /xmlHandler (e.g http:/proxy-host:8082/xmlHandler).
Xml queries need to be sent with content-type: text/xml (if no charset is specified UTF-8 is always assumed).
No url encoding or base64 encoding should be used.
Login can use standard HTTP basic auth or, in situations where that isn't practical, the following xml-based login
can be used instead:
(login request, client -> proxy)
(successful login reply for an admin user, proxy -> client)
The user identity is any user defined by the current usermanager.
A session id is by default valid until the proxy is restarted (no timeout).
Incorrect user/pass results in:
(failed login reply, proxy -> client)
Once a session id has been obtained, the syntax for status comma$$$ looks like this:
(proxy-status command, client -> proxy)
Multiple status comma$$$ can be sent in one request, like this:
(multiple comma$$$, client -> proxy)
To specify parameters for the status comma$$$, just add attributes:
(cws-connector command with name parameter, client -> proxy)
A control command uses a slightly different syntax, and there can be only one command per request.
(reset-connector control command, client -> proxy)
Command method 2 - HTTP get
-------------------
Uses the same url endpoint as post: /xmlHandler
Login is standard HTTP basic auth only (a session cookie called JSESSIONID will be set if supported).
The same comma$$$ are available, but everything is specified on the query string as follows:
http://proxy-host:8082/xmlHandler?comma ... herValue...
Examples:
/xmlHandler?command=proxy-users&profile=cable (retrieve all user sessions in profile cable)
/xmlHandler?command=cws-connectors&name=testconn (retrieve connector named testconn)
Control comma$$$ work the same way:
/xmlHandler?comand=osd-message&text=hello&name=userx (send osd text hello to all sessions for userx)
Example status web
------------------
The proxy comes with a simple web site (cs-status.war) made up entirely of client side javascript and xslt. This is
not a conventional web application, but rather a standalone script that will fetch xml from the proxy (ajax-style) and
format it using xslt directly in the browser. It has been tested in firefox 2/3, ie 6/7/8 and safari 3/4 (and probably
won't work in anything else). To get an idea of how the status web works with the xml api, check out the /api-test.html
test page.
To modify the visual appearance (or even structure) of the status web, look at the following files:
/xslt/cws-status-resp.xml - This is the xslt xml that transforms all status command responses into html (or xhtml).
It generates markup that will reference the css styles in: /css/style.css
For more information on xslt: http://en.wikipedia.org/wiki/XSL_Transformations
/js/cs-status.js - The javascript that defines the different sections, and handles pre-processing of the xml (before
the xslt transform) and post-processing of the html. The post-processing involves adding script handlers to
specific id's in the markup created by the xslt transform, making the status web somewhat interactive.
Adding javascript handlers to the browser dom tree (rather than hard coding them into the markup) is sometimes
referred to as 'unobtrusive' javascript.
The cs-status.war file (a zip containing the directory tree served by the built in httpd) can be replaced at runtime
and changes will take effect immediately, no restart required.